Image device based on interactive multimedia technology
By designing an imaging device based on interactive multimedia technology and combining deep learning models to analyze video and audio data, the problems of low diagnostic efficiency and data security in existing psychotherapy are solved, and a more efficient and safe psychological treatment process is achieved.
Patent Information
- Application Number
- CN202411895241.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-21
- Publication Date
- 2025-04-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
During the existing psychological treatment process, doctors need to focus highly, resulting in inefficient diagnosis and treatment, and the risk of information leakage in data management and monitoring.
An imaging device based on interactive multimedia technology is designed, including a display device, a human-computer interaction module, a data management module, a doctor-patient communication module and a monitoring module. It transmits diagnostic and treatment data through an internal network to avoid leakage, and uses deep learning models to analyze video and audio data to assist psychologists in diagnosis.
It improves the diagnostic accuracy and work efficiency of psychological treatment, reduces the intensity of doctors, and effectively protects patient information and avoids data leakage.
Smart Images
Figure CN119811598A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of psychotherapy, in particular to an imaging device based on interactive multimedia technology. Background Art
[0002] Mental illness refers to a phenomenon in which a person's thinking, emotions and behavior deviate from the social norms due to mental stress and interference. The greater the degree of deviation from social norms in psychology and behavior, the more serious the mental illness. Long-term excessive stress may lead to a series of mental illnesses, and being in this state for a long time may cause mental illness.
[0003] Usually, the psychotherapy process is through contact and communication between the psychologist and the patient, judging the patient's mental state through the facial expressions and body movements produced during the communication. In this process, the form of one question and one answer or one question and multiple answers is usually adopted. On the one hand, the psychologist must pay attention to the patient's answer, and on the other hand, he must also pay attention to the patient's body movements and facial expressions. This requires a high degree of concentration, which will quickly consume the doctor's physical strength, thereby reducing the efficiency of diagnosis and treatment. Summary of the invention
[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an imaging device based on interactive multimedia technology to solve the problems existing in the above-mentioned background technology.
[0005] The present invention provides the following technical solution: an imaging device based on interactive multimedia technology, comprising a display device, wherein the display device is provided with a human-computer interaction module, a data management module, a doctor-patient communication module, a monitoring module and a network module, the human-computer interaction module comprises an information verification unit and a detection unit, the data management module comprises a login unit and a database unit, and the doctor-patient communication module comprises a registration unit, a condition communication unit and a diagnosis record unit.
[0006] Furthermore, the display device is divided into two types, a doctor side and a patient side. The display device on the patient side is set in the hall, with multiple devices set up and placed in an area. The doctor side is set inside the doctor's office, and the display devices on the doctor side and the patient side are all-in-one computers.
[0007] Furthermore, the information verification unit includes an identity entry area and an identity verification area. The information entry unit is mainly used to enter the patient's identity information. The information that needs to be entered in the identity entry area includes name, contact information and a brief description of the condition. The identity verification area is provided with an ID card recognition device for uploading identity information. The detection unit is arranged inside each doctor's office for detecting the current doctor's status, including diagnosis, idle and away. These three states correspond to yellow, green and red lights respectively.
[0008] Furthermore, the network module is divided into an internal network and an external network. The data transmission between the data management system and the monitoring module is realized through the internal network. The internal network has a fast transmission speed and there is no need to worry about data leakage. The doctor-patient communication module transmits data through the external network, which is the Internet.
[0009] Furthermore, the database unit is internally connected to the doctor-patient communication module and the monitoring module, and internally records the data of the doctor-patient communication module and the monitoring module, including diagnosis time data, doctor-patient communication data and diagnosis data. The login unit is used to log in to the patient's personal account in the hospital's internal network, and can view previous diagnosis results and diagnosis process data.
[0010] Furthermore, the registration unit mainly recommends a suitable doctor based on the severity of the patient's condition. The doctor's photo and name are displayed inside, and a status light is set below the name. The color of the light corresponds to the current working status of the doctor.
[0011] Furthermore, the condition communication unit is provided with two parts: a diagnosis community and a communication community. The diagnosis community is mainly used as a communication platform between doctors and patients, and the communication community is used as a communication platform between patients.
[0012] Furthermore, the diagnosis record unit is mainly used to display the diagnosis record, including the diagnosis time, diagnosis result and treatment method.
[0013] Furthermore, the monitoring module includes a sound recording device, a video recording device and an analysis and processing unit. The sound recording device is used to record the voice of the conversation during psychotherapy, distinguish the voice between the doctor and the patient by the timbre, and convert the voice into text information, which is convenient for recording. The video recording device is used to record the patient's actions during the diagnosis process. The analysis and processing unit is used to analyze the data information recorded in the sound recording device and the video recording device, so as to obtain the machine diagnosis result.
[0014] Before analysis, the video data needs to be preprocessed, including denoising, cropping, scaling, etc., to remove a large amount of redundant information contained in the video data, such as background noise, irrelevant objects, etc., and then the video data is converted into an image matrix, and the audio data is converted into Mel-frequency cepstral coefficient (MFCC) features.
[0015] The deep learning model automatically extracts key features from video frames through convolutional neural networks (CNN), which may include color, shape, edges, facial expressions, and body movements. For video sequence data, the model also uses recurrent neural networks (RNN) to capture the temporal dependencies between frames, thereby extracting motion and contextual information in the video.
[0016] Use annotated video datasets to train the deep learning model. Select a certain video dataset and have a psychologist annotate the facial expressions, body movements and corresponding psychological states of the characters. The video dataset is stored in the database unit as a basis for comparison. During the training process, the model will learn how to map the input video data to predefined output categories or regression values. Through the back propagation algorithm and gradient descent method, the weights and bias parameters in the model are continuously adjusted to minimize the difference between the predicted values and the actual values. The trained model is used to analyze the facial expressions and body movements of the characters in the video to obtain the current psychological state of the character.
[0017] The audio data is converted into Mel-frequency cepstral coefficient (MFCC) features, and a high-pass filter is used to increase the energy of the high-frequency part to balance the spectrum. The filter formula y(t)=x(t)-αx(t-1) is used, where α is 0.97. The audio signal is then divided into multiple small segments (frames). Each frame usually contains 20 to 40 milliseconds of data. There is usually a certain overlap between adjacent frames, usually 50%, to ensure the continuity of information, and a window function (such as a Hamming window) is applied to each frame signal to reduce spectrum leakage.
[0018] The windowed signal is subjected to a fast Fourier transform (FFT) to obtain the spectrum of each frame. FFT converts the time domain signal into a frequency domain signal, thereby revealing the energy distribution of different frequency components in the signal. The spectrum is then mapped to the Mel frequency scale. The Mel frequency scale is a frequency scale that simulates human ear perception. It models the sensitivity of the human ear to sounds of different frequencies. The spectrum is then weighted averaged through a set of triangular filters, each filter corresponding to a Mel frequency band. Finally, the logarithm of each value output by the filter bank is taken to simulate the human ear's perception of sound intensity. The logarithmic Mel spectrum is logarithmically compressed and discrete cosine transformed (DCT) to obtain Mel frequency cepstral coefficients (MFCC). The purpose of DCT is to compress the spectrum to fewer coefficients and remove correlations. Usually the first 12 to 13 coefficients are retained because low-order coefficients contain the main spectral information and high-order coefficients are often related to noise.
[0019] A long short-term memory network (LSTM) suitable for sound analysis is selected. The model can capture the timing and spectral features in speech and is suitable for tasks such as emotion recognition. Then, a large amount of speech data labeled with psychological states (such as anger, happiness, sadness, etc.) is used to train the model. During the training process, the model will learn to extract information related to the psychological state from the MFCC features. The performance of the model is then evaluated on the test set, and adjustments and optimizations are made based on the results. This includes adjusting the model structure, hyperparameters, and training strategies to improve the accuracy and robustness of the model. Finally, the model is used to detect the psychological state of the person through sound recognition.
[0020] The recognition results of the video dataset and the sound dataset are combined in chronological order. The diagnosis process is in the form of question and answer, combining the patient's behavior and sound to more accurately judge the patient's psychology. This is used as the basis for machine judgment to assist psychologists to better grasp the patient's psychological state, thereby providing more accurate diagnosis and treatment reports.
[0021] Beneficial effects:
[0022] 1. This imaging device based on interactive multimedia technology is equipped with a data management system and a monitoring module, which are connected to the inside of the display device through an internal network, which can prevent the leakage of psychological diagnosis and treatment data and better protect patient information.
[0023] 2. This imaging device based on interactive multimedia technology is equipped with a monitoring module, which contains a sound recording device, a video recording device and an analysis and processing unit. It assists psychologists in diagnosis and treatment by analyzing the patient's voice, body movements and expressions, thereby improving work efficiency and diagnostic accuracy and reducing work intensity. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is a schematic diagram of the module structure of an imaging device based on interactive multimedia technology proposed by the present invention;
[0025] Figure 2 This is a flowchart of an image device auxiliary process based on interactive multimedia technology proposed by the present invention. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] Embodiment 1
[0028] See also Figure 1 An imaging device based on interactive multimedia technology includes a display device. The display device is provided with a human-computer interaction module, a data management module, a doctor-patient communication module, a monitoring module and a network module. The human-computer interaction module includes an information verification unit and a detection unit. The data management module includes a login unit and a database unit. The doctor-patient communication module includes a registration unit, a condition communication unit and a diagnosis record unit.
[0029] There are two types of display devices: the doctor side and the patient side. The patient side display device is set up in the hall, with multiple devices set up and placed in an area. The doctor side is set up inside the doctor's office. The display devices on the doctor side and the patient side are all-in-one computers.
[0030] The information verification unit includes an identity entry area and an identity verification area. The information entry unit is mainly used to enter the patient's identity information. The information that needs to be entered in the identity entry area includes name, contact information and a brief description of the condition. The identity verification area is equipped with an ID card recognition device for uploading identity information. The detection unit is set inside each doctor's office to detect the current doctor's status, including diagnosis, idle and away. These three states correspond to yellow, green and red lights respectively.
[0031] The network module is divided into internal network and external network. The data transmission between the data management system and the monitoring module is realized through the internal network, and the doctor-patient communication module transmits data through the external network, which is the Internet.
[0032] The database unit is internally connected to the doctor-patient communication module and the monitoring module, and internally records the data of the doctor-patient communication module and the monitoring module, including diagnosis time data, doctor-patient communication data and diagnosis data. The login unit is used to log in to the patient's personal account in the hospital's internal network, and can view previous diagnosis results and diagnosis process data.
[0033] The registration unit mainly recommends suitable doctors based on the severity of the patient's condition. The doctor's photo and name are displayed inside, and there is a status light under the name. The color of the light corresponds to the current working status of the doctor.
[0034] The medical condition communication unit consists of two parts: the diagnosis community and the communication community. The diagnosis community is mainly used as a communication platform between doctors and patients, while the communication community is used as a communication platform between patients.
[0035] The diagnosis record unit is mainly used to display the diagnosis record, including the diagnosis time, diagnosis result and treatment method.
[0036] Embodiment 2
[0037] See also Figure 1-Figure 2 The monitoring module includes a sound recording device, a video recording device and an analysis and processing unit. The sound recording device is used to record the voice of the conversation during psychotherapy, distinguish the voices between the doctor and the patient by timbre, and convert the voice into text information, which is convenient for recording. The video recording device is used to record the patient's actions during the diagnosis process. The analysis and processing unit is used to analyze the data information recorded in the sound recording device and the video recording device, so as to obtain the machine diagnosis result.
[0038] Before analysis, the video data needs to be preprocessed, including denoising, cropping, scaling, etc., to remove a large amount of redundant information contained in the video data, such as background noise, irrelevant objects, etc., and then the video data is converted into an image matrix, and the audio data is converted into Mel-frequency cepstral coefficient (MFCC) features.
[0039] The deep learning model automatically extracts key features from video frames through convolutional neural networks (CNN), which may include color, shape, edges, facial expressions, and body movements. For video sequence data, the model also uses recurrent neural networks (RNN) to capture the temporal dependencies between frames, thereby extracting motion and contextual information in the video.
[0040] Use annotated video datasets to train the deep learning model. Select a certain video dataset and have a psychologist annotate the facial expressions, body movements and corresponding psychological states of the characters. The video dataset is stored in the database unit as a basis for comparison. During the training process, the model will learn how to map the input video data to predefined output categories or regression values. Through the back propagation algorithm and gradient descent method, the weights and bias parameters in the model are continuously adjusted to minimize the difference between the predicted values and the actual values. The trained model is used to analyze the facial expressions and body movements of the characters in the video to obtain the current psychological state of the character.
[0041] The audio data is converted into Mel-frequency cepstral coefficient (MFCC) features, and a high-pass filter is used to increase the energy of the high-frequency part to balance the spectrum. The filter formula y(t)=x(t)-αx(t-1) is used, where α is 0.97. The audio signal is then divided into multiple small segments (frames). Each frame usually contains 20 to 40 milliseconds of data. There is usually a certain overlap between adjacent frames, usually 50%, to ensure the continuity of information, and a window function (such as a Hamming window) is applied to each frame signal to reduce spectrum leakage.
[0042] The windowed signal is subjected to a fast Fourier transform (FFT) to obtain the spectrum of each frame. FFT converts the time domain signal into a frequency domain signal, thereby revealing the energy distribution of different frequency components in the signal. The spectrum is then mapped to the Mel frequency scale. The Mel frequency scale is a frequency scale that simulates human ear perception. It models the sensitivity of the human ear to sounds of different frequencies. The spectrum is then weighted averaged through a set of triangular filters, each filter corresponding to a Mel frequency band. Finally, the logarithm of each value output by the filter bank is taken to simulate the human ear's perception of sound intensity. The logarithmic Mel spectrum is logarithmically compressed and discrete cosine transformed (DCT) to obtain Mel frequency cepstral coefficients (MFCC). The purpose of DCT is to compress the spectrum to fewer coefficients and remove correlations. Usually the first 12 to 13 coefficients are retained because low-order coefficients contain the main spectral information and high-order coefficients are often related to noise.
[0043] A long short-term memory network (LSTM) suitable for sound analysis is selected. The model can capture the timing and spectral features in speech and is suitable for tasks such as emotion recognition. Then, a large amount of speech data labeled with psychological states (such as anger, happiness, sadness, etc.) is used to train the model. During the training process, the model will learn to extract information related to the psychological state from the MFCC features. The performance of the model is then evaluated on the test set, and adjustments and optimizations are made based on the results. This includes adjusting the model structure, hyperparameters, and training strategies to improve the accuracy and robustness of the model. Finally, the model is used to detect the psychological state of the person through sound recognition.
[0044] The recognition results of the video dataset and the sound dataset are combined in chronological order. The diagnosis process is in the form of one question and one answer or one question and multiple answers. The patient's behavior and sound are combined to more accurately judge the patient's psychology. This is used as the basis for machine judgment to assist psychologists to better grasp the patient's psychological state, thereby providing more accurate diagnosis and treatment reports.
[0045] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An imaging device based on interactive multimedia technology, characterized in that: It includes a display device, which is provided with a human-computer interaction module, a data management module, a doctor-patient communication module, a monitoring module and a network module. The human-computer interaction module includes an information verification unit and a detection unit, the data management module includes a login unit and a database unit, and the doctor-patient communication module includes a registration unit, a condition communication unit, and a diagnosis record unit.
2. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The display device is of two types, the doctor side and the patient side. The patient side display device is set in the hall, with multiple devices set up and placed in an area. The doctor side is set inside the doctor's office. The display devices on the doctor side and the patient side are all-in-one computers.
3. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The information verification unit includes an identity entry area and an identity verification area. The information entry unit is mainly used to enter the patient's identity information. The information that needs to be entered in the identity entry area includes name, contact information and a brief description of the condition. The identity verification area is provided with an ID card recognition device for uploading identity information. The detection unit is arranged inside each doctor's office to detect the current doctor's status, including diagnosis, idle and away. These three states correspond to yellow, green and red lights respectively.
4. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The network module is divided into an internal network and an external network. The data transmission between the data management system and the monitoring module is realized through the internal network, and the doctor-patient communication module transmits data through the external network, which is the Internet.
5. The imaging device based on interactive multimedia technology according to claim 4, characterized in that: The database unit is internally connected to the doctor-patient communication module and the monitoring module, and internally records the data of the doctor-patient communication module and the monitoring module, including diagnosis time data, doctor-patient communication data and diagnosis data. The login unit is used to log in to the patient's personal account in the hospital's internal network, and can view previous diagnosis results and diagnosis process data.
6. The imaging device based on interactive multimedia technology according to claim 3, characterized in that: The registration unit mainly recommends a suitable doctor based on the severity of the patient's condition. The doctor's photo and name are displayed inside, and a status light is set below the name. The color of the light corresponds to the current working status of the doctor.
7. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The condition communication unit is provided with two parts: a diagnosis community and a communication community. The diagnosis community is mainly used as a communication platform between doctors and patients, and the communication community is used as a communication platform between patients.
8. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The diagnosis record unit is mainly used to display the diagnosis record, including the diagnosis time, diagnosis result and treatment method.
9. The imaging device based on interactive multimedia technology according to claim 1, characterized in that: The monitoring module includes a sound recording device, a video recording device and an analysis and processing unit. The sound recording device is used to record the voice of the conversation during psychotherapy, distinguish the voice between the doctor and the patient by the timbre, and convert the voice into text information. The video recording device is used to record the patient's actions during the diagnosis process. The analysis and processing unit is used to analyze the data information recorded in the sound recording device and the video recording device to obtain the machine diagnosis result.