An intelligent medical care pillow control method based on AI voice interaction and a pillow

Abstract

The application discloses an intelligent medical nursing pillow control method and pillow based on AI voice interaction and belongs to the technical field of intelligent nursing; the method comprises the following steps: synchronously collecting audio data and facial image data of a user; performing pronunciation abnormal voice type identification on the audio data, selecting an adaptive model to perform voice recognition, performing lip reading on the facial image data, fusing the voice recognition result and the lip reading result in a cross-modal mode to determine a final voice instruction; performing information integrity judgment on the final voice instruction, generating a follow-up question information to collect supplementary information when there is an information gap; and controlling a nursing execution module to perform massage, temperature control heating, air regulation and health monitoring operations according to the voice instruction and the supplementary information; and the application realizes reliable, accurate and emotionally warm intelligent nursing.

Description

Technical Field

[0001] This invention relates to the field of intelligent nursing technology, specifically to an intelligent medical nursing pillow control method and pillow based on AI voice interaction. Background Technology

[0002] Currently, the field of smart nursing pillows has accumulated certain technological expertise. In terms of interaction methods, there are already technical solutions that use voice recognition to control on / off operation and mode switching, with some products supporting voice wake-up words. In terms of physiological monitoring, there are technologies that use non-invasive methods such as piezoelectric sensors and infrared sensors to collect users' sleep physiological parameters such as heart rate, respiration, and body movement. In terms of massage therapy, there are already products that integrate vibration motors or airbag structures inside the pillow to achieve neck massage or pillow height adjustment. In terms of sleep aid and companionship, there are technologies that use built-in speakers to play white noise, sleep-aiding music, or provide simple voice broadcasts.

[0003] However, existing smart nursing pillows and related voice interaction control technologies still have the following shortcomings:

[0004] (1) The existing voice interaction of smart nursing pillows is mainly based on pure audio recognition. In nursing scenarios, they often face continuous noise from medical equipment such as ventilators and monitors, as well as sound interference in the environment of multiple people in the same room, which leads to a significant decrease in the accuracy of voice command recognition. Moreover, the voice recognition effect is not good for users with abnormal pronunciation such as unclear pronunciation, slow speech speed, and insufficient breath, making it difficult to meet the reliable interaction needs in actual nursing scenarios.

[0005] (2) The existing smart nursing pillows mainly use passive response mode. The system only responds to the clear instructions issued by the user and lacks the ability to actively ask questions based on the integrity of information. When the user's expression is vague or the information is incomplete, it cannot actively guide the user to supplement key information, which reduces the efficiency of interaction and the accuracy of nursing services.

[0006] (3) Existing smart nursing pillows lack the ability to effectively perceive and proactively care for users' emotional state. They cannot comprehensively judge users' psychological state based on their voice characteristics and physiological parameters, nor can they proactively initiate caring dialogues and provide personalized emotional responses when users are in negative emotions such as anxiety or loneliness. It is difficult to achieve a holistic nursing effect that integrates physical and mental health. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides an intelligent medical care pillow control method and pillow based on AI voice interaction, solving the problems mentioned in the background section.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution: a smart medical care pillow control method based on AI voice interaction, comprising the following steps:

[0011] S1 collects the user's audio data through a microphone array and simultaneously collects the user's facial image data through a camera;

[0012] S2, extract acoustic features from the audio data, identify abnormal speech types based on the acoustic features, select the corresponding model from multiple adapted speech recognition models based on the identified abnormal speech types, and obtain speech recognition results; extract the lip region from the facial image data and perform lip reading recognition to obtain lip reading recognition results; perform cross-modal fusion of the speech recognition results and lip reading recognition results to determine the final voice command;

[0013] S3, perform semantic understanding on the final voice command and extract the information elements already contained, compare the information elements already contained with the preset complete information element set to identify information gaps, if there are information gaps, generate follow-up questions based on the information gaps and initiate follow-up questions to the user through voice broadcast to collect supplementary information, if the information is complete, proceed to step S4.

[0014] S4. Generate a control command based on the final voice command and the supplementary information, and send the control command to the nursing execution module. The control command is used to control at least one of the massage operation, temperature control heating operation, air inflation adjustment operation and health monitoring operation.

[0015] S5, during the execution of the control command, the nursing execution module collects the user's physiological parameters in real time and continuously monitors the user's voice characteristics through the microphone array;

[0016] S6. Based on the voice features and physiological parameters, determine the user's current emotional state. When a preset proactive care trigger condition is detected, initiate a care dialogue with the user through voice broadcast.

[0017] In step S2, the first confidence score of each keyword in the speech recognition result is weighted and the second confidence score of the corresponding keyword in the lip reading result is weighted to obtain the weighted confidence score of each keyword. The final voice command is determined based on the weighted confidence score. The formula for calculating the weighted confidence score is as follows:

[0018] ;

[0019] in, This indicates the weighted confidence level of the keywords. This represents the first confidence level of the keyword in the speech recognition result, with a value ranging from 0 to 1. This represents the second confidence level of the keyword in the lip-reading results, with a value ranging from 0 to 1. The first fusion weight represents the speech recognition result. The second fusion weight represents the lip-reading recognition result, and both values ​​range from 0 to 1, and satisfy the following conditions: .

[0020] In step S2, the first fusion weight and the second fusion weight Based on the current environmental signal-to-noise ratio Dynamic calculation, The signal-to-noise ratio (SNR) of the current environment is expressed in decibels.

[0021] when hour, and ;when hour, and ;when hour, and Calculate according to the following formula:

[0022] ;

[0023] in, Indicates the first threshold. Indicates the second threshold, and When a user is detected to be unable to speak, force settings are applied. and .

[0024] The abnormal speech type identification in step S2 specifically includes: extracting formant frequency parameters, speech rate parameters, pitch fluctuation parameters, and volume attenuation parameters from the acoustic features; inputting the formant frequency parameters, speech rate parameters, pitch fluctuation parameters, and volume attenuation parameters into a pre-trained classification model, and outputting the user's abnormal speech type, which includes unclear pronunciation, slow speech rate, and insufficient breath; selecting the corresponding adapted speech recognition model according to the abnormal speech type, and enhancing the signal components of the preset frequency band in the acoustic features.

[0025] In step S3, the follow-up information is generated based on the user's historical dialogue data and the current dialogue context; the follow-up information includes follow-up questions about the nature of the user's discomfort symptoms, follow-up questions about the duration of symptoms, and follow-up questions about factors that aggravate or alleviate symptoms; the complete set of information elements includes chief complaint information elements, present medical history information elements, and past medical history information elements.

[0026] In step S6, the user's current emotional state is determined based on the speech features and the physiological parameters. Specifically, this includes: extracting speech rate parameters, pitch parameters, tremor parameters, and volume parameters from the speech features; extracting heart rate variability data and sleep quality data from the physiological parameters; inputting the speech rate parameters, pitch parameters, tremor parameters, volume parameters, heart rate variability data, and sleep quality data into a pre-trained emotion recognition model, and outputting the user's current emotional state category and corresponding emotion confidence score. The emotional state categories include anxiety, loneliness, calmness, and irritability.

[0027] The proactive care triggering conditions include any one of the following: the user's emotional state is identified as the anxiety state or the loneliness state and the duration exceeds the first preset duration; the user's sleep quality continuously declines for more than the second preset number of days; the user's voice interaction frequency is lower than the third preset threshold and the duration exceeds the fourth preset duration.

[0028] In step S6, when the caring dialogue is initiated, the user selects the corresponding mode from the emotion-driven response mode and the rational response mode based on the user's current emotional state, and retrieves the user's historical emotional memory data to generate personalized response content; the historical emotional memory data includes the user's historical emotional state records and the user's historical dialogue topic records.

[0029] A pillow, using the above-described control method, includes:

[0030] Pillow body;

[0031] The main control module is located inside the pillow body;

[0032] A voice interaction module is disposed within the pillow body. The voice interaction module includes a microphone array and a voice broadcasting unit. The microphone array is used to collect the user's audio data, and the voice broadcasting unit is used to broadcast voice information to the user.

[0033] A camera is embedded in the front edge of the pillow body to collect facial image data of the user;

[0034] A multimodal fusion module is electrically connected to the voice interaction module and the camera, respectively, and is used to perform cross-modal fusion of voice recognition results and lip reading results to determine the final voice command;

[0035] An abnormal pronunciation speech adaptive module is electrically connected to the speech interaction module and is used to identify the abnormal pronunciation speech type based on the acoustic features of the audio data, and select the corresponding adaptive speech recognition model according to the identified abnormal pronunciation speech type.

[0036] An active inquiry module, electrically connected to the multimodal fusion module, is used to perform semantic understanding on the final voice command to determine the completeness of the information, and to generate follow-up questions when the information is incomplete.

[0037] The psychological state perception module is electrically connected to the voice interaction module and the nursing execution module, respectively. It is used to determine the user's current emotional state based on the voice characteristics and the physiological parameters collected by the nursing execution module, and to trigger an active care dialogue when a preset active care trigger condition is detected.

[0038] The nursing execution module is electrically connected to the main control module and is located inside the pillow body. It is used to perform massage operation, temperature control heating operation, inflation adjustment operation and health monitoring operation.

[0039] The main control module is electrically connected to the voice interaction module, camera, multimodal fusion module, abnormal pronunciation voice adaptive module, active inquiry module, psychological state perception module, and nursing execution module.

[0040] (III) Beneficial Effects

[0041] (1) Audio data and facial image data are collected synchronously by microphone array and camera. The audio data is used to identify abnormal speech types and automatically switch the adaptation model. At the same time, the speech recognition results and lip reading results are fused across modally. The fusion weight of speech recognition and lip reading is dynamically allocated according to the environmental signal-to-noise ratio. When the signal-to-noise ratio is low or the user cannot speak, lip reading is completely relied upon. This achieves highly reliable voice interaction in noisy medical care scenarios, ensuring that users with abnormal speech such as unclear pronunciation, slow speech speed, and insufficient breath, as well as patients who cannot speak after intubation, can use it normally.

[0042] (2) The active inquiry module performs semantic understanding and information integrity judgment on voice commands. When the information is incomplete, it automatically generates follow-up questions that conform to the logic of medical consultation based on the large language model, guiding users to supplement key information such as the nature of symptoms, duration, and factors that aggravate or relieve symptoms. This improves the efficiency of interaction and the accuracy of nursing services, enabling users to obtain accurate nursing responses even if their expressions are vague.

[0043] (3) By integrating speech characteristics such as speech rate, pitch, tremor, and volume parameters, as well as physiological parameters such as heart rate variability and sleep quality data, the psychological state perception module comprehensively judges the user's emotional state. When it detects that the user is anxious, lonely, or has a continuous decline in sleep quality, it actively initiates a care dialogue and selects an emotion-driven or rational-driven response mode according to the emotional state. It retrieves the user's historical emotional memory to generate a personalized response, thus achieving an upgrade from passive response to proactive care and achieving the technical effect of holistic care for mind and body. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0045] Figure 2 This is a system structure block diagram of the various modules inside the pillow body in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figures 1 to 2 As shown, the embodiments of the present invention provide the following technical solutions:

[0048] A pillow using the following control method, please refer to [link / reference]. Figure 2 Specifically, it includes:

[0049] Pillow body;

[0050] The main control module is located inside the pillow body;

[0051] The voice interaction module is located inside the pillow body. The voice interaction module includes a microphone array and a voice broadcasting unit. The microphone array is used to collect the user's audio data, and the voice broadcasting unit is used to broadcast voice information to the user.

[0052] A camera is embedded in and fixedly connected to the front edge of the pillow body to collect facial image data of the user.

[0053] The multimodal fusion module is electrically connected to the voice interaction module and the camera, respectively, and is used to perform cross-modal fusion of the voice recognition result and the lip reading result to determine the final voice command;

[0054] The abnormal pronunciation speech adaptive module is electrically connected to the speech interaction module. It is used to identify the abnormal pronunciation speech type based on the acoustic features of the audio data, and select the corresponding adaptive speech recognition model according to the identified abnormal pronunciation speech type.

[0055] The active inquiry module, electrically connected to the multimodal fusion module, is used to perform semantic understanding on the final voice command to determine the completeness of the information, and to generate follow-up questions when the information is incomplete.

[0056] The psychological state perception module is electrically connected to the voice interaction module and the nursing execution module, respectively. It is used to determine the user's current emotional state based on voice characteristics and physiological parameters collected by the nursing execution module, and to trigger proactive care dialogue when a preset proactive care trigger condition is detected.

[0057] The nursing execution module, electrically connected to the main control module, is located inside the pillow body and is used to perform massage operations, temperature control and heating operations, inflation adjustment operations, and health monitoring operations.

[0058] The main control module is electrically connected to the voice interaction module, camera, multimodal fusion module, abnormal pronunciation voice adaptive module, active inquiry module, psychological state perception module, and nursing execution module.

[0059] A method for controlling an intelligent medical care pillow based on AI voice interaction; please refer to [link / reference]. Figure 1 This includes the following steps:

[0060] S1 collects the user's audio data through a microphone array and simultaneously collects the user's facial image data through a camera;

[0061] After the user issues a voice command, the microphone array collects audio data of the user's environment in real time. This audio data includes the user's voice signal and any ambient noise. Simultaneously, the camera activates upon being triggered by a voice wake-up signal, synchronously acquiring image data of the user's face. The microphone array employs a multi-microphone configuration and supports beamforming technology, enabling sound source localization and directional sound pickup to enhance the user's voice signal and suppress ambient noise from other directions. Both audio and facial image data are transmitted digitally to the main control module for processing by subsequent modules. During the acquisition process, the microphone array continuously collects audio data, while the camera only operates during each user wake-up interaction, automatically shutting down after the interaction ends. The camera only initiates image acquisition upon detecting user voice wake-up; otherwise, it remains completely off. The acquired facial image data is processed locally by the multimodal fusion module, which performs lip region extraction and lip-reading recognition. After processing, the original image data and intermediate processed data are destroyed and not uploaded to any external devices or cloud servers.

[0062] S2. Extract acoustic features from audio data, identify abnormal speech types based on the acoustic features, select the corresponding model from multiple adapted speech recognition models based on the identified abnormal speech types, and obtain speech recognition results; extract the lip region from facial image data and perform lip reading recognition to obtain lip reading recognition results; perform cross-modal fusion of speech recognition results and lip reading recognition results to determine the final speech command.

[0063] After receiving the audio data and facial image data, the main control module performs speech recognition processing and lip-reading recognition processing respectively, specifically as follows:

[0064] Speech recognition processing: The voice interaction module extracts acoustic features from the audio data and transmits these features to the speech abnormality adaptation module. The speech abnormality adaptation module first identifies the speech abnormality type based on the acoustic features; specifically, it extracts formant frequency parameters, speech rate parameters, pitch fluctuation parameters, and volume attenuation parameters from the acoustic features. Formant frequency parameters reflect the acoustic characteristics of the vocal tract structure during pronunciation, speech rate parameters reflect the speed of pronunciation, pitch fluctuation parameters reflect the amplitude of pitch variation, and volume attenuation parameters reflect the degree of volume reduction. These four parameters are input into a pre-trained classification model, which outputs the user's speech abnormality type. Speech abnormality types include unclear pronunciation, slow speech rate, and insufficient breath support. Unclear pronunciation is characterized by unclear articulation and blurred syllable boundaries; slow speech rate is characterized by a significantly slower-than-normal speech rate; and insufficient breath support is characterized by insufficient volume and weak breath. Based on the identified speech abnormality type, the corresponding model is selected from multiple adaptive speech recognition models for speech recognition. The adaptive speech recognition models corresponding to different speech abnormality types are optimized and trained according to their respective pronunciation features. That is, the adaptation model for ambiguous pronunciation uses acoustic feature input with a longer time window to capture contextual information, the adaptation model for slow speech rate uses dynamic time warping algorithm to adapt to the user's slow pronunciation rhythm, and the adaptation model for insufficient breath uses low volume signal enhancement preprocessing and targeted training acoustic model.

[0065] Simultaneously, the signal components in the acoustic features within a preset frequency band are enhanced. This preset frequency band corresponds to the frequency range where energy attenuation is significant when the user's pronunciation is abnormal. The pronunciation abnormality speech adaptation module inputs the enhanced acoustic features into the selected adaptation model and outputs a speech recognition result with a first confidence level. The speech recognition result contains multiple keywords, each keyword corresponding to a first confidence level. The first confidence level ranges from 0 to 1 and is used to indicate the reliability of the keyword recognition result.

[0066] The lip-reading process involves a multimodal fusion module that extracts lip region images from facial image data captured by a camera. Specifically, it first locates the user's facial region using a face detection algorithm, then identifies key points of the lip contour based on a facial key point detection algorithm, and crops a sequence of lip region images. Since lip-reading relies on the temporal changes in lip movements, the multimodal fusion module organizes the lip region image sequence into a frame sequence according to time order. Subsequently, a temporal convolutional network is used to perform lip-reading on the lip region image sequence, extracting dynamic change features of lip movements. These dynamic change features are mapped to a text sequence, outputting a lip-reading result with a second confidence level. Each keyword in the lip-reading result corresponds to a second confidence level, with a value ranging from 0 to 1, representing the reliability of the lip-reading result for that keyword.

[0067] The multimodal fusion module performs a weighted calculation on the first confidence score of each keyword in the speech recognition result and the second confidence score of the corresponding keyword in the lip-reading result to obtain the weighted confidence score of each keyword. The final voice command is determined based on the weighted confidence score. The formula for calculating the weighted confidence score is as follows:

[0068] ;

[0069] in, This indicates the weighted confidence level of the keywords. This represents the first confidence level of the keyword in the speech recognition result, with a value ranging from 0 to 1. This represents the second confidence level of the keyword in the lip-reading results, with a value ranging from 0 to 1. The first fusion weight represents the speech recognition result. The second fusion weight represents the lip-reading recognition result, and both values ​​range from 0 to 1, and satisfy the following conditions: .

[0070] First fusion weight Second fusion weight Based on the current environmental signal-to-noise ratio Dynamic calculation, This represents the current environmental signal-to-noise ratio (SNR) in decibels (dB). The SNR is calculated from the ratio of speech signal energy to noise signal energy in the environmental audio signal collected by the microphone array. This indicates that the current environment is relatively quiet and the audio signal quality is high. and Completely trust the speech recognition results; when When the noise level is high, it indicates that the ambient noise is significant and the audio signal quality is poor. and Completely trust the lip-reading results; when When this time, it indicates that the current ambient noise level is moderate. and Calculate according to the following formula:

[0071] ;

[0072] in, Indicates the first threshold. Indicates the second threshold, and The values ​​of the first and second thresholds can be calibrated and adjusted according to the actual usage scenario, calculated through the above piecewise linear interpolation. and It can achieve a smooth transition when the signal-to-noise ratio is between two thresholds, avoiding sudden changes in recognition results caused by weight abrupt changes.

[0073] Furthermore, when the system detects that a user is unable to speak, such as when the user is unable to produce clear speech due to post-operative intubation, throat disease, or extreme weakness, the multimodal fusion module will force a setting. and This means that the final voice command is determined entirely by lip reading results. The inability to speak can be detected in several ways: if the microphone array does not detect a valid voice signal within a preset time while the camera simultaneously captures continuous lip movement, the user can be determined to be in a speechless state; or it can be manually triggered by a user-preset silent mode. Finally, the multimodal fusion module determines the final voice command based on the weighted confidence of each keyword.

[0074] S3, perform semantic understanding on the final voice command and extract the information elements already contained, compare the information elements already contained with the preset complete information element set to identify information gaps, if there are information gaps, generate follow-up questions based on the information gaps and initiate follow-up questions to the user through voice broadcast to collect supplementary information, if the information is complete, proceed to step S4.

[0075] After receiving the final voice command, the main control module performs semantic understanding on it using the active inquiry module. The active inquiry module employs a natural language understanding engine based on a large language model to perform intent recognition and information extraction on the final voice command. It extracts the information elements already contained in the user's voice command, based on preset semantic slots. These semantic slots define the information required to complete a specific care intent. For example, for a massage control intent, semantic slots might include the massage location, massage mode, and massage intensity; for a heat therapy control intent, semantic slots might include the heating level and heating duration. The active inquiry module compares the already contained information elements with a preset complete set of information elements. This complete set of information elements predetermines the minimum information elements required to effectively respond to a given user intent. If any missing elements exist between the already contained information elements and the complete set of information elements, an information gap is identified.

[0076] If an information gap exists, the proactive inquiry module generates follow-up questions based on the specific content of the gap. These follow-up questions are generated using the user's historical dialogue data and the current dialogue context, driven by a large language model to generate follow-up questions that conform to medical consultation logic and use plain and natural language. Follow-up questions include those concerning the nature of the user's symptoms, such as "Is it aches, stings, or stiffness?"; those concerning the duration of symptoms, such as "How long has this been going on?"; and those concerning factors that aggravate or alleviate symptoms, such as "Under what circumstances will it worsen? Under what circumstances will it ease?". The depth and frequency of follow-up questions are adaptively adjusted based on the user's state. That is, when a user's tone is detected as hurried or shows impatience, the number of follow-up questions is reduced or the content is simplified; when the user responds positively, the depth of follow-up questions can be appropriately increased to obtain more complete information. The voice broadcasting unit will broadcast follow-up information to the user in voice form. After the user answers the follow-up questions by voice, the active inquiry module continues to extract information from the user's answer, repeating the above comparison, identification of information gaps and follow-up questioning process until the collected supplementary information meets the requirements of a complete set of information elements, or the user actively terminates the inquiry.

[0077] If the information is complete, the active inquiry module will transmit the final voice command and the supplementary information collected throughout the questioning process to the main control module, and proceed to step S4.

[0078] S4. Generate control instructions based on the final voice command and supplementary information, and send control instructions to the nursing execution module. The control instructions are used to control at least one of the following operations: massage operation, temperature control heating operation, air inflation adjustment operation, and health monitoring operation.

[0079] The main control module generates corresponding control commands based on the final voice command and supplementary information collected through follow-up questioning, according to preset control logic. These control commands include, but are not limited to, massage control commands, temperature control commands, inflation adjustment commands, and health monitoring commands. The main control module then sends these generated control commands to the nursing execution module, which drives the corresponding execution components to perform the operation based on the command type. The nursing execution module can execute a single operation or multiple operations in parallel. For example, temperature control and health monitoring can be performed simultaneously during a massage, achieving coordinated operation of physiotherapy and monitoring. For instance, when the user command is "small pillow, cervical massage, soothing mode," the main control module sends a massage control command to the nursing execution module, driving the cervical massage component to operate in soothing mode. When the user command is "small pillow, turn on heat therapy, level 2," the main control module sends a temperature control command to the nursing execution module, driving the temperature control heating component to operate at the second temperature level.

[0080] S5, during the execution of control commands, collects the user's physiological parameters in real time through the nursing execution module and continuously monitors the user's voice characteristics through the microphone array;

[0081] During the execution of control commands by the nursing execution module, the built-in health monitoring sensor group collects the user's physiological parameters in real time. The health monitoring sensor group includes a piezoelectric thin-film sensor array, consisting of 256 flexible piezoelectric sensor nodes arranged in the middle of the pillow body and the neck support area. When the user lies on the pillow, the sensor array conforms to the pressure distribution of the user's head and neck. When the piezoelectric thin-film sensors are subjected to pressure changes, they generate an electric charge signal proportional to the rate of pressure change. This signal is amplified and converted from analog to digital to form a digitized pressure signal sequence. The main control module extracts the user's physiological parameters from the pressure signal sequence, including heart rate, respiratory rate, body movement amplitude, and number of turns. The heart rate parameter is obtained through spectral analysis of the periodic micro-vibration components in the pressure signal synchronized with the cardiac cycle; the respiratory rate parameter is obtained through spectral analysis of the low-frequency vibration components in the pressure signal synchronized with the respiratory cycle; and the body movement amplitude and number of turns are obtained by detecting and counting the components with abrupt changes in amplitude in the pressure signal.

[0082] Meanwhile, the voice interaction module continuously monitors the user's voice characteristics through a microphone array. Voice characteristics include parameters such as the user's speech rate, pitch, tremor, and volume. This voice feature monitoring differs from the voice recognition in step S2. Step S2 identifies and parses explicit voice commands actively issued by the user, while the voice feature monitoring in this step continuously collects and extracts acoustic parameters from the user's natural voice signal, without involving semantic understanding of the voice content. This is to reduce power consumption and protect user privacy. The microphone array operates continuously at a low sampling rate, only extracting acoustic feature parameters without complete voice recording.

[0083] S6 determines the user's current emotional state based on voice characteristics and physiological parameters. When a preset proactive care trigger condition is detected, it proactively initiates a care dialogue with the user through voice broadcast.

[0084] In step S6, the user's current emotional state is determined based on speech features and physiological parameters. Specifically, this includes: extracting speech rate parameters, pitch parameters, tremor parameters, and volume parameters from the speech features; extracting heart rate variability data and sleep quality data from the physiological parameters; inputting the speech rate parameters, pitch parameters, tremor parameters, volume parameters, heart rate variability data, and sleep quality data into a pre-trained emotion recognition model, and outputting the user's current emotional state category and corresponding emotion confidence. The emotional state categories include anxiety, loneliness, calmness, and irritability.

[0085] The psychological state perception module extracts heart rate variability data and sleep quality data from physiological parameters. Heart rate variability data is obtained by analyzing fluctuations between successive heartbeats; high-frequency components are related to parasympathetic nerve activity, and the ratio of low-frequency to high-frequency components reflects the balance between the sympathetic and parasympathetic nervous systems. Sleep quality data includes parameters such as deep sleep duration, light sleep duration, and number of awakenings, generated by the main control module after analyzing signals collected by the sensor array during the user's sleep. For example, a trend of shortened deep sleep duration and increased awakenings can comprehensively reflect the user's sleep recovery quality.

[0086] The psychological state perception module integrates the aforementioned speech rate, pitch, tremor, volume, heart rate variability, and sleep quality data, and inputs them into a pre-trained emotion recognition model. It then outputs the user's current emotional state category and corresponding emotion confidence score. The emotion recognition model employs a gradient-boosting decision tree-based classification model. Its training data comes from a multimodal physiological and speech joint dataset labeled with emotion tags. Emotional state categories include anxiety, loneliness, calmness, and agitation. Specifically, anxiety is characterized by an increased low-to-high frequency ratio of heart rate variability, faster speech rate, and higher tremor; loneliness is characterized by slower speech rate, lower pitch, and significantly reduced interaction frequency; calmness is characterized by parameters within the normal range of the user's baseline with minimal fluctuation; and agitation is characterized by faster speech rate, higher volume, and reduced heart rate variability.

[0087] The conditions for triggering proactive care include the following three situations:

[0088] The user's emotional state is identified as anxious or lonely and lasts for more than a first preset duration. This duration can be personalized according to individual user differences. When the user is in an anxious or lonely state for more than this duration, it indicates that the user may be in a persistent negative emotion and needs external intervention and care.

[0089] The user's sleep quality has declined for more than the second preset number of days. The criteria for judging the decline in sleep quality are that the daily sleep quality score is lower than the user's individual baseline value by a certain percentage, or the deep sleep duration is continuously reduced, the number of awakenings is continuously increased, etc. When the user's sleep quality continues to decline for more than this number of days, it indicates that the user may have cumulative physiological or psychological stress.

[0090] The user's voice interaction frequency is below the third preset threshold and the duration exceeds the fourth preset duration. Voice interaction frequency refers to the number of times a user initiates voice interaction per unit of time. When the user's interaction frequency is significantly lower than their daily level and lasts for a long time, it may indicate that the user is in a state of social withdrawal or low mood.

[0091] When any one of the three proactive care triggering conditions mentioned above is met, the psychological state perception module determines that proactive care needs to be initiated and sends a proactive care trigger signal to the voice interaction module. The voice interaction module then proactively initiates a care dialogue with the user through the voice broadcast unit. The content of the care dialogue is generated based on the user's current emotional state; that is, the psychological state perception module transmits the identified emotional state category and its confidence level, as well as the specific conditions triggering proactive care, to the voice interaction module.

[0092] In step S6, when initiating a caring dialogue, the voice interaction module selects a response mode based on the user's current emotional state. Response modes include an emotion-driven response mode and a rational response mode. When the user is anxious or agitated, the emotion-driven response mode is selected. In this mode, the response focuses on emotional comfort and empathy, using a gentle and sensitive language style to prioritize alleviating the user's emotional distress, with information provision taking a secondary role. When the user is lonely, the emotion-driven response mode is selected. In this mode, the response focuses on proactive companionship and topic guidance, creating a sense of companionship through open-ended questions and sharing-based dialogue, prioritizing the establishment of an emotional connection.

[0093] When the user is in a calm state and proactive care is triggered by decreased sleep quality or reduced interaction frequency, the rational-led response mode is selected. In this mode, the response content focuses on health data analysis and objective suggestions, with a rational and clear language style, and prioritizes providing actionable health improvement suggestions.

[0094] Meanwhile, when generating specific responses, the voice interaction module retrieves user historical emotional memory data stored in the psychological state perception module. This historical emotional memory data includes records of the user's historical emotional states and historical dialogue topics. For example, when the system detects that the user is currently in a state of loneliness, the voice broadcast unit can proactively broadcast: "You haven't been talking much lately. Is something bothering you, or do you simply want some quiet time? I'm always here; just call me if you want to chat." This response not only utilizes the user's historical emotional memory but also includes specific nursing care suggestions, achieving an organic integration of emotional care and nursing services.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0096] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0097] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling an intelligent medical care pillow based on AI voice interaction, characterized in that, Includes the following steps: S1 collects the user's audio data through a microphone array and simultaneously collects the user's facial image data through a camera; S2, extract acoustic features from the audio data, identify abnormal speech types based on the acoustic features, select the corresponding model from multiple adapted speech recognition models based on the identified abnormal speech types, and obtain the speech recognition result; The lip region is extracted from the facial image data and lip reading is performed to obtain the lip reading result; The speech recognition results and lip reading results are fused across modalities to determine the final speech command; S3, perform semantic understanding on the final voice command and extract the information elements already contained, compare the information elements already contained with the preset complete information element set to identify information gaps, if there are information gaps, generate follow-up questions based on the information gaps and initiate follow-up questions to the user through voice broadcast to collect supplementary information, if the information is complete, proceed to step S4. S4. Generate a control command based on the final voice command and the supplementary information, and send the control command to the nursing execution module. The control command is used to control at least one of the massage operation, temperature control heating operation, air inflation adjustment operation and health monitoring operation. S5, during the execution of the control command, the nursing execution module collects the user's physiological parameters in real time and continuously monitors the user's voice characteristics through the microphone array; S6. Based on the voice features and physiological parameters, determine the user's current emotional state. When a preset proactive care trigger condition is detected, initiate a care dialogue with the user through voice broadcast.

2. The intelligent medical care pillow control method based on AI voice interaction according to claim 1, characterized in that, In step S2, the first confidence score of each keyword in the speech recognition result is weighted and the second confidence score of the corresponding keyword in the lip reading result is weighted to obtain the weighted confidence score of each keyword. The final voice command is determined based on the weighted confidence score. The formula for calculating the weighted confidence score is as follows: ； in, This indicates the weighted confidence level of the keywords. This represents the first confidence level of the keyword in the speech recognition result, with a value ranging from 0 to 1. This represents the second confidence level of the keyword in the lip-reading results, with a value ranging from 0 to 1. The first fusion weight represents the speech recognition result. The second fusion weight represents the lip-reading recognition result, and both values ​​range from 0 to 1, and satisfy the following conditions: .

3. The intelligent medical care pillow control method based on AI voice interaction according to claim 2, characterized in that, In step S2, the first fusion weight and the second fusion weight Based on the current environmental signal-to-noise ratio Dynamic calculation, The signal-to-noise ratio (SNR) of the current environment is expressed in decibels. when hour, and ;when hour, and ;when hour, and Calculate according to the following formula: ； in, Indicates the first threshold. Indicates the second threshold, and When a user is detected to be unable to speak, force settings are applied. and .

4. The intelligent medical care pillow control method based on AI voice interaction according to claim 1, characterized in that, The abnormal speech type identification in step S2 specifically includes: extracting formant frequency parameters, speech rate parameters, pitch fluctuation parameters, and volume attenuation parameters from the acoustic features; inputting the formant frequency parameters, speech rate parameters, pitch fluctuation parameters, and volume attenuation parameters into a pre-trained classification model, and outputting the user's abnormal speech type, which includes unclear pronunciation, slow speech rate, and insufficient breath; selecting the corresponding adapted speech recognition model according to the abnormal speech type, and enhancing the signal components of the preset frequency band in the acoustic features.

5. The intelligent medical care pillow control method based on AI voice interaction according to claim 1, characterized in that, In step S3, the follow-up information is generated based on the user's historical dialogue data and the current dialogue context; the follow-up information includes follow-up questions about the nature of the user's discomfort symptoms, follow-up questions about the duration of the symptoms, and follow-up questions about factors that aggravate or alleviate the symptoms. The complete set of information elements includes chief complaint information elements, present medical history information elements, and past medical history information elements.

6. The intelligent medical care pillow control method based on AI voice interaction according to claim 1, characterized in that, In step S6, the user's current emotional state is determined based on the speech features and the physiological parameters. Specifically, this includes: extracting speech rate parameters, pitch parameters, tremor parameters, and volume parameters from the speech features; extracting heart rate variability data and sleep quality data from the physiological parameters; inputting the speech rate parameters, pitch parameters, tremor parameters, volume parameters, heart rate variability data, and sleep quality data into a pre-trained emotion recognition model, and outputting the user's current emotional state category and corresponding emotion confidence score. The emotional state categories include anxiety, loneliness, calmness, and irritability.

7. The intelligent medical care pillow control method based on AI voice interaction according to claim 6, characterized in that, The proactive care triggering conditions include any one of the following: the user's emotional state is identified as the anxiety state or the loneliness state and the duration exceeds the first preset duration; the user's sleep quality continuously declines for more than the second preset number of days; the user's voice interaction frequency is lower than the third preset threshold and the duration exceeds the fourth preset duration.

8. The intelligent medical care pillow control method based on AI voice interaction according to claim 1, characterized in that, In step S6, when the caring dialogue is initiated, the user selects the corresponding mode from the emotion-driven response mode and the rational response mode based on the user's current emotional state, and retrieves the user's historical emotional memory data to generate personalized response content; the historical emotional memory data includes the user's historical emotional state records and the user's historical dialogue topic records.

9. A pillow, using the AI ​​voice interaction-based intelligent medical care pillow control method as described in claim 1, characterized in that, include: Pillow body; The main control module is located inside the pillow body; A voice interaction module is disposed within the pillow body. The voice interaction module includes a microphone array and a voice broadcasting unit. The microphone array is used to collect the user's audio data, and the voice broadcasting unit is used to broadcast voice information to the user. A camera is embedded in the front edge of the pillow body to collect facial image data of the user; A multimodal fusion module is electrically connected to the voice interaction module and the camera, respectively, and is used to perform cross-modal fusion of voice recognition results and lip reading results to determine the final voice command; An abnormal pronunciation speech adaptive module is electrically connected to the speech interaction module and is used to identify the abnormal pronunciation speech type based on the acoustic features of the audio data, and select the corresponding adaptive speech recognition model according to the identified abnormal pronunciation speech type. An active inquiry module, electrically connected to the multimodal fusion module, is used to perform semantic understanding on the final voice command to determine the completeness of the information, and to generate follow-up questions when the information is incomplete. The psychological state perception module is electrically connected to the voice interaction module and the nursing execution module, respectively. It is used to determine the user's current emotional state based on the voice characteristics and the physiological parameters collected by the nursing execution module, and to trigger an active care dialogue when a preset active care trigger condition is detected. The nursing execution module is electrically connected to the main control module and is located inside the pillow body. It is used to perform massage operation, temperature control heating operation, inflation adjustment operation and health monitoring operation. The main control module is electrically connected to the voice interaction module, camera, multimodal fusion module, abnormal pronunciation voice adaptive module, active inquiry module, psychological state perception module, and nursing execution module.

Images