On-device voice-based AI health mate system and its operating method

Abstract

The present invention relates to a voice-based health mate that provides health management and emotional support using an on-device voice intelligence module for the elderly and people with developmental disabilities in rural and remote areas where internet access is not smooth. The present invention integrates speech recognition, language understanding, and speech synthesis into a single AST engine and incorporates a microphone, speaker, and local database to process the user's natural language speech completely offline. The local SLM module utilizes a healthcare knowledge base and a vector search engine to generate personalized responses in real-time, such as medication confirmation, biometric data recording, and conversational consultation, while the anomaly detection module detects abnormal speech patterns or schedule deviations and immediately sends notifications to guardians and medical institutions. The scheduling agent provides voice notifications for medication, exercise, and test appointments, and the MQTT communication module connects with external organizations while maintaining data integrity.

Description

Technology Field

[0001] The present invention relates to a voice-based AI health mate system that utilizes on-device voice recognition, small language model (SLM), and text-to-speech (TTS) technologies to provide health status checks, emotional support, detection of abnormal signs, and automatic notification services for the elderly and people with developmental disabilities, even in environments where the internet connection is unstable. Background Technology

[0002] Korea is already transitioning from an aging society to a super-aging society, and particularly in rural and fishing communities, over 30% of the total population consists of people aged 65 or older. The proportion of elderly people living alone in these areas is significantly higher than in cities, making it easy to miss the "golden hour" for receiving external assistance in the event of an emergency. At the same time, with approximately 40% of the elderly suffering from chronic diseases, the demand for routine health monitoring and emotional care is rapidly increasing. However, in rural and remote areas, internet infrastructure is often unstable or disconnected, making it difficult to use cloud-based healthcare services or smartphone applications in real time. Furthermore, due to the low accessibility and proficiency of the elderly regarding digital devices, it is difficult to provide sufficient user convenience through complex touch interfaces or text-based services.

[0003] Voice-based interfaces enable natural interaction while minimizing eye and hand limitations, making them particularly useful for digitally vulnerable groups such as the elderly, individuals with developmental disabilities, and the visually impaired. However, conventional speech recognition and synthesis systems rely on cloud computing resources, leading to issues such as high network latency and the risk of personal information leakage at data transmission points. While On-Device Automatic Speech-SLM-TTS technology is garnering attention as an alternative to address these problems, providing high-quality voice services in real-time within a low-power environment requires the integrated implementation of lightweight models, local databases, event detection logic, and protocols for connecting with external systems.

[0004] Registered Patent No. 10-2426745 describes a system designed to improve the mental health and dietary habits of the elderly through counselor support. It includes a counseling scenario provision unit, an abnormal sign detection unit, a mental health diagnosis unit, a counseling content display unit, a counselor evaluation unit, and a food recommendation determination unit. This technology features a counselor-centric structure that integrates various functions, such as customized question distribution, counseling quality control based on multi-layered verification, analysis of intimacy between the elderly and the counselor, and food recommendations. It is significant in that it assesses the risk of mental illness and suggests individual health management directions based on collected data.

[0005] However, the aforementioned conventional technology presupposes a scenario design and result interpretation process involving the intervention of professional counselors, which limits its application in environments where counselors are unavailable or in real-time emergency response situations. Since it utilizes text- and graphic-based counselor terminals as the core interface, it is difficult for the elderly to use independently; furthermore, because data transmission between the user terminal and the server is essential, usability is poor in areas with unstable networks. While it focuses on mental health and dietary management, it lacks the ability to immediately detect and notify physical vital signs or acute abnormal situations such as falls and cardiac arrest via voice. Additionally, the failure to utilize on-device voice models presents limitations in terms of personal information protection and latency, and automatic notification functions to various linked agencies (public health centers, 119 emergency services, family members, caregivers, etc.) are not sufficiently implemented.

[0006] Due to the limitations of such conventional technologies, rapid and continuous health management is difficult in environments with low accessibility to professional personnel and unstable networks, such as for elderly people living alone in rural areas or individuals with developmental disabilities. Therefore, there is a need for an integrated solution that operates based on a local voice model regardless of internet connectivity, checks health status through everyday conversation, and automatically sends alerts to registered affiliated organizations immediately upon the detection of abnormal signs. Furthermore, there is an urgent need for the development of technology capable of providing emotional support through direct interaction with users without the need for counselors, and delivering personalized information by combining various detection sensors, scheduling agents, and vector search engines. In response to these needs, a voice-based Health Mate system needs to be established that integrates on-device AST technology, abnormal signal detection logic, schedule-based agents, local vector search, and MQTT-based linkage protocols. The problem to be solved

[0007] This system implements a fully voice-interface-based health management service utilizing on-device voice recognition and synthesis technology, even in environments with unstable internet connections, to efficiently monitor the health status and provide emotional support to the elderly, including those residing in rural and remote areas. Furthermore, it supports rapid response to emergencies by automatically transmitting alerts to registered relevant institutions when abnormal signs are detected, and aims to expand emotional and health care for vulnerable groups, such as children and adults with developmental disabilities. means of solving the problem

[0008] To achieve the above objective, the present invention provides an on-device voice health mate system designed to operate even when the internet is unstable or disconnected. The system receives and outputs user speech from a voice input / output unit composed of a microphone and a speaker, and an ASR module immediately converts the received voice signal into text. The converted text is transmitted to an SLM module to generate conversational response sentences and health assessment results through natural language understanding and generation and context combination based on a vector search engine. The vector search engine linked with the SLM module searches the vector store for user conversation history and medical information that have been segmented and embedded by a data chunking module to reinforce real-time context. The generated response sentence is converted into speech by a TTS module and output through a speaker.

[0009] This system maintains a latency-free voice interface even in unstable network environments by keeping pre-trained ASR, SLM, and TTS models resident in local memory, and automatically restarts the relevant module or switches to a backup model if performance degradation is detected. The healthcare model and anomaly detection module analyze voice content, vital signs, and schedule compliance information to detect abnormal signs exceeding thresholds, and immediately send notification messages containing hash-based integrity verification values ​​to pre-designated medical institutions and guardians via an MQTT broker-based communication module. The scheduling agent proactively calls users based on personalized health and medication schedules recorded in the database, and triggers an alert in conjunction with the anomaly detection module if there is a prolonged lack of response.

[0010] When a user profile for developmental disabilities is activated, the SLM module automatically adjusts pragmatic complexity, vocabulary level, and utterance length, while the TTS module regulates emotional richness and utterance speed to reduce cognitive burden. The communication module ensures data integrity by including a hash value for each packet and executes retransmission logic if an acknowledgment is not received. Furthermore, the system monitors speech recognition accuracy, TTS accuracy, response generation speed, and agent call accuracy in real-time, and executes automatic recovery procedures if performance drops below a critical threshold. All voice and text data, embeddings, and logs are stored in the database and vector store for future use in diagnosis and model improvement.

[0011] The voice-based health mate system of the present invention configured in this manner ensures both user safety and convenience by integrating real-time voice conversation, personalized health management, and automatic detection and notification of abnormal signs in an on-device environment. Effects of the invention

[0012] This invention significantly improves accessibility and reliability compared to existing internet-based services by enabling easy health status checks for the elderly and individuals with developmental disabilities through an on-device voice interface, even in rural and remote environments where internet connectivity is unstable. By implementing core technologies such as speech recognition, natural language processing, and speech synthesis locally, personal information protection and data security are enhanced, while crisis response capabilities are improved through real-time emergency detection and automatic notifications to affiliated organizations. Furthermore, it contributes to the psychological stability and quality of life of users by providing emotional support and personalized health management services. High service reliability can be guaranteed based on data integrity, response speed, and the accuracy of schedule execution. Brief explanation of the drawing

[0013] FIG. 1 is a block diagram of an on-device voice-based health mate system configuration; FIG. 2 is a flowchart for the system operation method of FIG. 1; FIG. 3 is an example diagram showing the summary screen of the system dashboard; Figure 4 is an example diagram showing a list of individual monitoring data. Specific details for implementing the invention

[0014] The configuration and operation of a specific embodiment according to the present invention will be described in detail with reference to the drawings.

[0015] The present invention comprises an integrated system comprising a voice input / output unit (100), an ASR module (110), an SLM module (120), a vector search engine (130), a TTS module (140), a healthcare model (150), an abnormal signal detection module (160), a scheduling agent (170), an MQTT broker-based communication module (180), a database (190), a vector store (200), a data chunking module (210), and a performance monitoring module (220) for implementing a voice-based healthcare service in an on-device environment.

[0016] Referring to FIGS. 1 and 2, the voice input / output unit (100) collects and outputs voice signals through a microphone and a speaker, and the collected voice is transmitted to an ASR (Automatic Speech Recognition) module (110). The ASR module (110) converts the input voice signal into text and provides it to a SLM (Small Language Model) module (120). The SLM module (120) generates text embeddings to understand the intent and context of the user's utterance and matches the embeddings with past conversations or domain knowledge stored in a vector store (200) through a vector search engine (130). The vector search engine (130) supports the generation of an accurate response by extracting the most relevant vector and returning it to the SLM module (120).

[0017] The response text generated by the SLM module (120) is synthesized into natural speech by the TTS (Text To Speech) module (140) and provided to the user through the voice input / output unit (100). At the same time, the healthcare model (150) evaluates the user's health status by analyzing biometric information, speech patterns, schedule information, etc. extracted during the response generation process. The evaluation results are transmitted to the abnormal signal detection module (160), and the abnormal signal detection module (160) detects abnormal signs exceeding a predefined threshold in real time. When an abnormality is detected, an immediate notification is sent to a medical institution or guardian via the MQTT (Message Queuing Telemetry Transport) broker-based communication module (180).

[0018] The schedule agent (170) retrieves healthcare schedule data registered by the user, such as medication intake, exercise, and regular checkups, from the database (190) and triggers the SLM module (120) to be called at a designated time. During this process, the data chunking module (210) divides the large-capacity schedule data into chunks considering storage efficiency and stores them in the vector store (200). The MQTT broker-based communication module (180) is responsible for message exchange between the on-device system and an external linked organization, and minimizes data integrity and transmission delay.

[0019] The database (190) stores user profiles, speech logs, healthcare history, etc., in a structured form, and the vector store (200) manages text embeddings, voice embeddings, and metadata in a high-dimensional vector form. The performance monitoring module (220) collects and analyzes key indicators such as ASR accuracy, TTS accuracy, response delay, and data integrity in real time to support maintaining the overall quality of the system.

[0020] The voice-based health mate system according to the present invention configured as described above processes the entire process on-device even in an environment where the internet is unstable, thereby enabling the elderly and people with developmental disabilities to use the service stably.

[0021] The above voice input / output unit (100) includes a microphone array, a speaker, a touch button, a status indicator LED, and a wireless communication antenna.

[0022] The microphone array collects the user's speech while minimizing ambient noise by evenly arranging omnidirectional microphones in a cylindrical shape. The collected analog voice signal passes through a low-noise amplifier, is converted into a digital signal, and is transmitted to the ASR module (110). The speaker clearly reproduces the synthesized voice output from the TTS module (140).

[0023] Touch buttons provide physical inputs such as power, volume, and call keys, while it is desirable that major interactions be handled via voice. Status indicator LEDs visualize operating states, such as system standby, voice reception, anomaly detection, and network transmission, using color and flashing patterns; it is also desirable to provide the same information via voice for users with visual impairments.

[0024] The wireless communication antenna is linked with the MQTT broker-based communication module (180). It is desirable to provide an external antenna connector to prepare for cases where signal attenuation is severe in rural and indoor environments. In the event of a communication line failure, a notification packet is temporarily stored in a local buffer memory and automatically transmitted upon reconnection.

[0025] The voice input / output unit (100) also provides a vital sensor expansion port linked to a healthcare model (150), allowing blood pressure, pulse, and oxygen saturation modules to be connected in a plug-and-play manner. Data from the sensors is recorded in a database (190) along with timestamps, and it is desirable to analyze them in real time by an abnormal signal detection module (160).

[0026] Overall, the voice input / output unit (100) integrates voice, visual, tactile, and wireless interfaces to serve as a hub that unifies user speech input, synthesized voice output, system status display, external device connection, and remote communication functions.

[0027] The above ASR module (110) plays the role of converting a voice signal input through the voice input / output unit (100) into a character sequence in real time and transmitting it to the SLM module (120) and subsequent stages. It is desirable that the above ASR module be designed to enable completely offline operation, taking into account the speech patterns unique to the elderly or users with developmental disabilities and the irregular communication conditions of rural environments.

[0028] The ASR module (110) is structured in layers, including a front-end preprocessing pipeline, acoustic model subblocks, pronunciation dictionary, language model cache, decoder, adaptor, etc.

[0029] The preprocessing pipeline performs automatic gain control, noise reduction, and speech activity detection on the raw signal collected from the microphone array to improve the accuracy of feature extraction in the subsequent stage. The feature extractor converts the preprocessed signal into a Mel-Spectrogram or MFCC series vector, and the converted frame is input into an acoustic model.

[0030] The acoustic model receives frames generated by feature extraction and performs inference on the NPU / DSP. In other words, it plays the role of encoding meaningful representations and probabilistic signals of speech so that the decoder can read them, in order to connect acoustic patterns to characters, phonemes, and subwords.

[0031] The pronunciation dictionary is constructed to include standard Korean pronunciation, major dialect variations, onomatopoeia and mimetic words, and non-standard intonation patterns, and the language model cache is preferably updated with real-time cumulative statistics of term frequency and context specialized for the healthcare domain to dynamically correct probabilities during decoding.

[0032] The decoder outputs results immediately in token units. That is, it generates a sequence of tokens that become text. This result is first immediately transmitted to the SLM module (120) to configure a prompt for generating a conversational response, and second, reported to the performance monitoring module (220) along with the probability and CER (Character Error Rate) prediction values ​​to self-diagnose the model quality.

[0033] The above ASR module (110) constantly monitors emergency keywords (“save me,” “119,” “it hurts,” etc.) using an internal attention map of the encoder or a separate hotword list. If the detection reliability by the abnormal signal detection module and healthcare model described later exceeds a threshold value, it is desirable to push an event to the abnormal signal detection module (160) regardless of the SLM module (120) to immediately trigger an MQTT notification sequence.

[0034] Parameter updates of the ASR module are performed based on field voice logs accumulated in the data chunking module (210) and the vector store (200), and it is desirable that the scheduling agent (170) is managed to execute a fine-tuning pipeline during off-peak hours.

[0035] The above SLM module (120) is responsible for generating dialogue for an on-device voice-based health mate system. The module preferably incorporates a lightweight language model of the transformer family and consists of the following sub-configurations.

[0036] First, the input preprocessing unit tokenizes the text output by the ASR module (110) and reconstructs the conversation history.

[0037] Second, the context binding unit transmits the query generated by the input preprocessing unit to the vector search engine (130) to calculate the similarity with the domain knowledge fragment stored in the vector store (200). The search results are rearranged along with the original utterance in the order of “system prompt → domain knowledge → user utterance” to form an input token sequence of the language model. The system prompt may include a safety guide, and the domain knowledge may include materials such as past conversations or psychological counseling content.

[0038] The above context coupling unit first fixes the safety and role using a system prompt, then lays the groundwork with domain knowledge, and finally attaches a user utterance so that the SLM model generates a grounded response while guarding the safety and policy guardrails.

[0039] Third, it is desirable for the generation engine to reduce TTS conversion delay by applying a streaming inference method and immediately starting output on a token basis.

[0040] Fourth, the post-processing and censorship unit links the generated response text with the abnormal signal detection module (160) to determine in the first instance whether it is profanity or dangerous speech. If a risk factor is detected, it is desirable to immediately call the schedule agent (170) and send a warning message to a pre-registered contact through the MQTT broker-based communication module (180).

[0041] Fifth, the schedule comparison unit compares and determines the time and action slots of the generated response candidates with the list of scheduled events retrieved from the schedule agent (170).

[0042] If a match or conflict is detected, the response automatically includes a 'user reminder' or 'additional question (suggestion to register / change / delete)', and requests the scheduling agent to create, modify, or cancel the corresponding event if necessary.

[0043] The above SLM module (120) serves as a hub for the vector search data flow within the ASR-SLM-TTS pipeline. That is, when a user utterance is input, data moves in the order of ① ASR result → ② vector search → ③ SLM prompt combination → ④ response generation → ⑤ TTS conversion.

[0044] Meanwhile, if the response generated by the SLM module contains predefined abnormal keywords such as “urgent” or “pain,” it is desirable for the abnormal signal detection module (160) to detect this and immediately send a notification to the relevant agency through the MQTT broker (180).

[0045] It is desirable for the above SLM module (120) to maintain an up-to-date knowledge base by reloading its own on-device parameters according to a signal from the periodic performance monitoring module (220) or by triggering an update of the contents of the vector store (200) when necessary.

[0046] The vector search engine (130) is a module that rapidly connects domain knowledge and current queries (user utterance context) necessary for generating conversations, notifications, and status questions of a voice-based healthcare system, and performs knowledge embedding, indexing, searching, re-ranking, and context assembly in a single pipeline.

[0047] A voice-based healthcare system first collects domain knowledge through short message cards, protocol documents, medication guides, FAQs, device setup instructions, and emergency response checklists, and segments it into passage units (e.g., 256 to 512 characters) after undergoing normalization tailored to the characteristics of the Korean language (spelling / spacing correction, number / unit standardization, abbreviation expansion, and synonym mapping). Each passage is converted into a fixed-length vector through an embedding encoder and is assigned metadata (topic, risk level, recency, access rights, and user group).

[0048] In the query flow, query embeddings are created by combining recent conversation summaries, intents / slots, and context tags (medication / falls / difficulty breathing, etc.) with the ASR transcript obtained through microphone input.

[0049] The vector search engine generates a context bundle including best similarity, grounds span (exact location within the document), metadata, and confidence metrics, which is inserted into SLM prompts or used directly for selecting status check question templates via speakers.

[0050] Similarity primarily uses cosine similarity, and a guardrail threshold is applied that reflects false positive / missed detection statistics collected during operation. If the maximum similarity is below the threshold, it is routed to the “Information Insufficient / Clarification Needed” path to output more general closed-ended questions or follow-up questions; if it is above the threshold, it proceeds to reasoned responses, customized guidance, or emergency protocol mapping.

[0051] The vector search engine continuously reports search success rates, re-ranking conversion rates, context adoption rates, and response satisfaction signals (user confirmation, subsequent error status) to the performance monitoring module, and thresholds, weights, and cache policies are automatically adjusted based on the data.

[0052] The above vector search engine is equipped to provide evidence most relevant to the user's current utterance context with low latency, and to reliably guide downstream actions such as status check questions, emergency alerts, and medication guidance when necessary.

[0053] The above TTS module (140) serves to rapidly convert text transmitted from the SLM module (120) and the vector search engine (130) into a voice signal and output it to the speaker of the voice input / output unit (100). It is desirable for the TTS module to keep all synthesis parameters and model weights resident in the terminal's internal storage device so that the voice synthesis function can operate continuously even in rural or remote environments where the internet connection is unstable.

[0054] The TTS module (140) can be subdivided into a preprocessing unit, a pronunciation and intonation prediction unit, an acoustic model unit, a vocoder unit, and a caching unit.

[0055] The preprocessing unit segments the input text into morphemes and phonemes and generates a sequence of phonetic symbols. The pronunciation and intonation prediction unit selects intonation patterns appropriate for each conversational situation, such as health counseling, emotional comfort, and notification guidance. The acoustic model unit receives the selected intonation information and phonetic symbol sequence to generate an intermediate representation in the form of a Mel-spectrogram, and the vocoder unit restores this to the final PCM speech. The caching unit reduces response delay by reusing the Mel-spectrogram or PCM when the same phrase is requested repeatedly.

[0056] It is preferable for the TTS module (140) to process the response text output by the SLM module (120) in a real-time streaming manner and start partial synthesis before the text is fully received. This allows the overall system target response delay to be met.

[0057] For example, the TTS tone is based on a mid-to-low range female speaker tone familiar to elderly users, but it is desirable to configure it to apply a softer intonation profile when the healthcare model (150) determines the emotional state as 'depressed'. The intonation profile is realized through a conditional conversion layer within the acoustic model section.

[0058] As another example, when the abnormal signal detection module (160) determines an emergency situation, the scheduling agent (170) transmits the keyword 'emergency notification scenario' to the TTS module (140). It is desirable for the TTS module (140) to minimize notification transmission delay by immediately retrieving a predefined short and clear voice warning from the caching unit and bypassing the synthesis process.

[0059] When a user inputs a health question by voice, the ASR module (110) converts it into text and transmits it to the SLM (120), then the TTS module (140) responds by voice, and in the final step, the process of recording the performance monitoring results in the database (190) is repeated.

[0060] The above healthcare model (150) evaluates the user's health status based on the recognition results transmitted from the ASR module (110) and generates customized feedback by organically cooperating with the SLM module (120), vector search engine (130), and TTS module (140).

[0061] The healthcare model (150) first includes a text-based intent detection submodule to map a user's utterance to a medical domain ontology. This submodule extracts multiple intent and attribute values ​​in real time, such as "medication time reminder," "complaining of pain," and "request for emotional support," and transmits the extraction results to a scheduling agent (170) in the form of structured JSON.

[0062] For example, attribute values ​​corresponding to the intent of 'medication time reminder' could be 'drug name, dosage, time of administration,' etc., while attribute values ​​corresponding to the intent of 'complaining of pain' could be 'location, intensity, duration,' etc. Additionally, attribute values ​​for 'request for emotional support' could be 'emotion classification, triggering factor,' etc.

[0063] When a user says, "Remind me to take two Tylenol pills at 6 PM. My back has been hurting a lot since yesterday," this voice is converted into text by the ASR module, and the healthcare model selects candidates through a text classifier and validates and reinforces them through ontology mapping to extract the intent as "medication time reminder" and "complaint of pain," and extracts the corresponding attribute values ​​as "Tylenol, 2 tablets, 18:00" and "back, a lot," respectively, generates them in the form of structured JSON, and then transmits them to the schedule agent.

[0064] Second, an acoustic-based biosignal estimation submodule is provided to analyze the frequency spectrum, formants, and breathing cycle of the microphone input, thereby calculating patterns such as coughing, shortness of breath, and voice tremors in real time. The acoustic-based biosignal estimation submodule is serially connected to an abnormal signal detection module (160), so that when patterns such as coughing, shortness of breath, and voice tremors are calculated, the indicators (coughing, shortness of breath, voice tremors, etc.) are transmitted to the abnormal signal detection module. When the abnormal signal detection module performs a final abnormality judgment and a risk level is detected, an alarm message is immediately sent to pre-registered family members, local government health centers, and emergency rescue centers via an MQTT broker-based communication module (180).

[0065] Third, the user's age, chronic disease history, recent medication history, etc., including a personalized health management algorithm, are retrieved from the database (190), and a daily morning medication check routine or a weekly depression check routine is automatically generated. The routine is triggered by a schedule agent (170), and it is desirable for the healthcare model (150) to generate an interaction script in natural language form by utilizing the conversation context of the SLM module (120).

[0066] Fourth, the user's emotional state is classified into 'Good, Caution, Danger' stages by comprehensively evaluating speech content, accent, and vocabulary usage frequency, including an emotion analysis submodule. If the classification result is a 'Danger' stage, appropriate comforting and encouraging phrases can be searched for in a vector store (200) where pre-built psychological counseling FAQs are stored via a vector search engine (130) and delivered to a TTS module (140).

[0067] Finally, it is desirable to configure the performance monitoring module (220) to continuously collect the CER, response delay, and notification success rates of the healthcare model (150) and to send logs to the administrator if they exceed a threshold. This allows the system to operate in a way that satisfies the requirement of '95% or more of data integrity between heterogeneous systems'.

[0068] The above abnormal signal detection module (160) calculates the risk level by comparing and analyzing in real time the biometric and voice parameters extracted from the healthcare model (150) and the individual reference values ​​stored in the database (190). The above abnormal signal detection module consists of a data normalization unit, an abnormal sign inference unit, a threshold adaptation unit, and a notification interface unit.

[0069] The data normalization unit receives multiple parameters, such as the frequency spectrum, breathing pattern, speech rate, heart rate, and oxygen saturation of the voice signal collected through the voice input / output unit (100), from the healthcare model (150) and maps them to a standard scale that reflects age, gender, and personal health history. At this time, it is desirable to also include linguistic features (e.g., negative emotion keywords, frequency of self-talk) extracted from the SLM module (120) in the normalization target.

[0070] The anomaly inference unit queries the vector store (200) with the normalized parameter vector to perform a similarity search, and then calculates a risk index by combining the cosine similarity and temporal rate of change with the suspected case. If the risk index exceeds a pre-set first threshold, it generates a candidate event, and it is desirable to determine the final anomaly by cross-validating with the Bayesian network of the healthcare model (150) in a second step.

[0071] The threshold adaptation unit automatically corrects individual and time-series thresholds based on false positive and false negative statistics received from the performance monitoring module (220). Through this, it is desirable to learn indicator deviations caused by voice tremors or chronic diseases unique to elderly users, thereby maintaining the CER target value even during long-term use.

[0072] The threshold adaptation unit aggregates the false positive and false negative rates by user and time period provided by the performance monitoring module using an exponential moving average, and calculates the threshold by weighting the false negative cost more heavily than the false positive cost (e.g., 2:1).

[0073] For example, if User A's false positive rate significantly exceeds the target (5%) in the nighttime period (18%), the threshold is raised from 0.65 to 0.74 to suppress the frequency of nighttime notifications, and if User B's missing rate rises to 22% in the period immediately after exercise, the threshold is lowered from 0.70 to 0.56 to reduce the possibility of missing emergencies.

[0074] The notification interface unit calls the MQTT broker-based communication module (180) to publish an event topic to a registered guardian terminal, local government care center, 119 emergency center, etc., when an abnormal signal occurs. For example, when the abnormal signal detection module (160) issues an “abnormal_detect” packet, the communication module (180) relays the message, and the receiving side returns an ACK.

[0075] The above-mentioned schedule agent (170) is a control unit that coordinates each module of the on-device voice health mate system to smoothly perform time-based events. The schedule agent periodically queries schedule information stored in the database (190) to update the event queue and calculates the trigger time by synchronizing with the real-time system clock.

[0076] The schedule agent receives schedule setting, modification, and deletion commands entered by the user in natural language via a voice interface in a structured form from the SLM module (120), parses them, and records them in a table format in the database. The schedules registered in this way are inserted into an event queue and managed along with metadata such as priority, urgency, and repetition status.

[0077] When the trigger time arrives, the scheduling agent first checks the current status flags of the abnormal signal detection module (160) and the healthcare model (150) to verify that there are no user safety issues, and then requests the TTS module (140) to generate a voice prompt. At the same time, it publishes a notification message to a topic of a guardian or local care agency via the MQTT broker-based communication module (180) so that local and remote notifications are provided in parallel.

[0078] It is desirable for the schedule agent to include a low-power timer, a non-volatile scheduling cache, and a rollback stack for recovery in the event of an error.

[0079] It is desirable for the schedule agent to automatically generate weekly and monthly reports, record them in a database, and, when necessary, send them to a vector search engine (130) so that they can be used for analyzing user behavior patterns. Through this, personalization policies such as long-term health management strategies or adjusting reminder frequencies can be dynamically updated.

[0080] The above MQTT broker-based communication module (180) is configured to mediate not only data transmission and reception between the voice input / output unit (100) and each internal module, but also message exchange with external linked organizations. Since the MQTT broker-based communication module utilizes MQTT, a publish-subscribe protocol, it enables reliable data transmission even in environments with unstable network quality, such as rural and mountainous regions.

[0081] When the above abnormal signal detection module (160) detects an abnormal pattern, the schedule agent (170) publishes an alarm topic, and the MQTT broker-based communication module (180) immediately relays it to the MQTT broker. The broker relays the message to the linked organization terminal that has subscribed in advance. At this time, the SLM module (120) that subscribes to the same topic itself also receives the message and can update the voice guidance scenario.

[0082] It is desirable for the above MQTT broker-based communication module (180) to process periodic health data transmission and event-based alarm transmission separately. For periodic transmission, it is desirable to minimize delay with QoS 0, and for emergency notifications, it is desirable to set QoS 2 so that a reception acknowledgment is received at least once.

[0083] Additionally, it is desirable to configure the MQTT broker-based communication module (180) to be linked with the database (190) to store MQTT packet metadata as a log and to transmit it to the performance monitoring module (220) to aggregate indicators such as transmission success rate and latency.

[0084] The above database (190) is a local storage layer that manages all structured and unstructured data generated and consumed by the on-device voice health mate system. This database can store original voice files collected from the voice voice input / output unit (100), text transcripts converted by the ASR module (110), conversation history of the SLM module (120), biosignal analysis results of the healthcare model (150), event logs of the abnormal signal detection module (160), medication and lifestyle pattern schedules of the schedule agent (170), and system metrics of the performance monitoring module (220) by separating them into individual tables.

[0085] The data chunking module (210) divides the transcribed text into semantic units and records the metadata (timestamp, speaker ID, etc.) of the corresponding chunk together in the database. Subsequently, when the embedding vector extracted from the SLM module is stored in the vector store (200), the database ensures traceability of the query results by maintaining a relational mapping with the original chunk by referencing the vector's primary key. When the vector search engine (130) returns the query results, it is desirable for the database to simultaneously load the related conversation and healthcare history to prevent loss of conversation context.

[0086] Data transmitted to an external medical and welfare institution through an MQTT broker-based communication module (180) is first recorded in a database in the form of a transaction log.

[0087] The database provides time-series indexes to analyze long-term trends in medical and health data, and also offers a visualization API to view the timing of abnormal signal occurrences and corresponding user status changes in graph form.

[0088] The performance monitoring module (220) serves as a control core for ‘visualizing and immediately correcting’ the quality and reliability of the on-device voice health mate system in real time, and performs the role of collecting, analyzing, recording, and providing feedback on events and metrics across the entire range from the voice input / output unit, ASR module, SLM module, vector search engine, TTS module, healthcare model, abnormal signal detection module, MQTT broker-based communication module, and database.

[0089] The operational goals are to maintain the quality of speech recognition and synthesis, minimize response delays, guarantee transmission reliability, and ensure data integrity between modules, and the results are reflected in the database on a periodic and event-based basis.

[0090] The performance monitoring module first collects key performance indicators in a standardized format. In the recognition layer, the ASR decoder reports token-level probabilities and CER prediction values ​​together, while in the synthesis layer, it aggregates TTS stability indicators (pronunciation and intonation prediction consistency, vocoder error rate, etc.) and response generation latency.

[0091] In addition, at the system level, it monitors data integrity, event correlations between modules, and the success of storage writes to detect quality degradation early in all stages of “collection, storage, and retrieval.”

[0092] These collected items include ASR accuracy, TTS accuracy, response latency, data integrity, etc.

[0093] Network and notification reliability are closely integrated with the MQTT broker-based communication module. Packet metadata generated by the communication module during the publish / subscribe process (ACK reception status, number of retries, transmission success rate by QoS, round-trip delay) is stream-collected by the performance monitoring module, which aggregates this data to calculate transmission success rates and latency metrics.

[0094] The performance monitoring module periodically collects false positive / miss positive statistics from the healthcare model and anomaly detection module to feed back for the automatic correction of individual and time-series thresholds. Through this, it learns indicator deviations caused by the characteristics of elderly users or chronic diseases to support the maintenance of target CERs and alert sensitivity even during long-term use.

[0095] From an operational perspective, administrators use dashboards and reports generated by the performance monitoring module to identify pipeline bottlenecks, areas with frequent recognition errors, transmission failure points, and variability by user group and time zone at a glance, and execute model fine-tuning, dictionary (pronunciation / ontology) augmentation, knowledge base re-indexing, and schedule policy adjustments as necessary.

[0096] When predefined triggers occur, such as the alarm level of the abnormal signal detection module, sudden fluctuations in sensor / microphone-based biomarkers, prolonged non-response or inactivity, the arrival of medication time, or regular inspection events by the schedule agent, the healthcare module prioritizes outputting a concise and misrecognition-resistant closed-ended question through the TTS module.

[0097] Questions can be configured to collect “yes / no” or “number / single word” responses. After the first question, a preset response waiting time is applied (e.g., 10 seconds), and if no response or an uncertain response is detected, a clarification question is repeated 2 to 3 times. If no response persists despite repetition or if the danger signal rises, the system immediately switches to an emergency simplification procedure to play a short confirmation, and then propagates an MQTT notification to registered guardians, local governments, or emergency centers.

[0098] If a sudden change is detected in breathing, coughing, or speaking patterns, the healthcare module first checks the status in the order of safety, breathing, and consciousness. For example, it asks, “Are you having trouble breathing right now? Please answer ‘Yes’ or ‘No.’” and if the response is “Yes,” it presents a follow-up question: “Do you have chest pain or severe dizziness?”

[0099] If the user responds "Yes," the message "Are you sitting or lying down in a safe position right now?" is immediately displayed to guide the first self-stabilization measures, and at the same time, the transmission of an emergency alert is initiated. If the user responds "No," the conversation is terminated with a short recovery confirmation message, such as "Please try breathing in and out slowly for a moment. Are you okay now?" or monitoring is extended.

[0100] When the medication reminder time has arrived but there is no medication log, the healthcare module asks, “Did you take your medication just now? Please answer with ‘I took it’ or ‘Not yet.’” If the response is “Not yet,” it confirms the intention to take the medication with the question, “Would you like to take it now? Please say ‘Now’ or ‘Later.’” If the answer is “Now,” it prints a brief medication instruction message and records the medication check. If the answer is “Later,” the scheduling agent updates the reminder time.

[0101] If persistent voice tremors or changes in speech rate suggestive of emotional stress, the healthcare module approaches the user with non-judgmental and empathetic language. For instance, it asks, “It seems you are feeling a bit tired lately. Could you describe how you are feeling in one word? Please select from ‘I’m okay,’ ‘I’m anxious,’ or ‘I’m depressed.’” If a negative response is received, it suggests a self-relief routine by saying, “Shall we take a short moment to regulate our breathing? If you need help, please say ‘Help.’” If the user responds with “Help,” it initiates the procedure for contacting a pre-set guardian or professional agency, accompanied by a guidance message.

[0102] In scenarios where ensuring safety is the top priority, concise and imperative sentences are used to enhance speed. For example, when irregular inhalation / exhalation and dizziness are simultaneously suspected, the message “Please sit and rest without moving right now. If you need help, please say ‘help me’” is output immediately, and confirmation questions are deferred to a later priority.

[0103] In all scenarios, the system records the conversation in the order of question → response → confirmation → action, and in failure or no-response conditions, it plays it back without delay using predefined alert and guidance messages.

[0104] FIG. 3 is an example diagram visually illustrating the monitoring request and processing status of a voice-based health mate system according to the present invention. As shown in the figure, the voice-based health mate system manages the total number of monitoring cases, as well as the waiting, processing, completed, and error statuses.

[0105] In addition, the number of requests over a certain period is displayed as a line graph to identify trends in fluctuations over time, and the request distribution by equipment is provided as a pie chart, allowing for intuitive verification of the load at the unit of specific devices (Device-001~Device-005).

[0106] In addition, processing status is aggregated into bar graphs categorized by waiting, in progress, completed, failed, and canceled states, allowing administrators to visualize overall service quality in real time.

[0107] Through this configuration, data linked to the performance monitoring module (220) is displayed in the form of a dashboard, and the user can intuitively diagnose bottlenecks, delays, and failures during operation.

[0108] FIG. 4 is an example diagram showing a list of individual monitoring requests of a voice-based health mate system according to the present invention.

[0109] As illustrated in the drawing, the monitoring screen is composed of items such as equipment ID, urgency, file name, processing status, and creation date and time, and the user can search by specific device or urgency conditions through the search bar. Each item is provided with a detailed view and editing function, allowing the administrator to immediately check the details of the voice file-based monitoring data and modify it if necessary. This screen configuration is implemented in a way that efficiently retrieves original voice files, transcribed text, and status logs stored in the database (190) at the user interface layer, and enables real-time updates by linking with the performance monitoring module (220) and the schedule agent (170). Through this, the user can quickly identify and respond to urgent items even amidst a large amount of request data.

[0110] Although the present invention has been described above with reference to embodiments thereof, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0111] 100: Voice Input / Output Unit 110: ASR Module 120 : SLM Module 130 : Vector Search Engine 140 : TTS Module 150 : Healthcare Model 160: Anomaly detection module 170: Schedule agent 180: MQTT broker-based communication module 190: Database 200 : Vector Store 210 : Data Chunking Module 220: Performance monitoring module

Claims

Claim 1 An on-device voice interface system that operates regardless of internet connection status, comprising: a voice input / output unit including a microphone and a speaker for receiving and outputting user speech; an ASR module for recognizing speech signals and converting them into text; an SLM module for generating natural language by receiving the recognition results of the ASR module; a TTS module for converting the output sentences of the SLM module into speech; a healthcare model for evaluating the user's health status; an abnormal signal detection module for identifying abnormal signs based on the evaluation results of the healthcare model; and an MQTT broker-based communication module for transmitting the abnormal signal identification results to internal modules or related organizations. A voice-based AI health mate system comprising a scheduling agent that coordinates time-based events, wherein the scheduling agent is configured to check the status flags of the abnormal signal detection module and the healthcare model at the time of triggering, request the TTS module to generate a prompt, and simultaneously issue a notification through the MQTT broker-based communication module, and wherein the ASR module is configured to constantly monitor urgent keywords and, when the detection reliability is above a threshold, push an event to the abnormal signal detection module regardless of the processing path of the SLM module to trigger an MQTT notification sequence. Claim 2 A voice-based AI health mate system according to claim 1, characterized in that the voice-based health mate system further includes a vector search engine that calculates similarity with domain knowledge based on the recognition result of the ASR module. Claim 3 A voice-based AI health mate system according to claim 2, characterized in that the vector search engine is configured to embed user conversation history and medical information data segmented through a data chunking module and store them in a vector store, and to search for the corresponding vector according to a query of an SLM module and reflect it in the generation of a response. Claim 4 A voice-based AI health mate system according to claim 1, wherein the voice-based health mate system further includes a scheduling agent for coordinating time-based events, and the scheduling agent is configured to request the generation of a TTS prompt and simultaneously issue an MQTT notification after checking the status flags of an abnormal signal detection module and a healthcare model at the time of the trigger. Claim 5 A voice-based AI health mate system according to claim 1, characterized in that the ASR module constantly monitors urgent keywords and, when the detection reliability is above a threshold, pushes an event to an abnormal signal detection module regardless of the SLM path to trigger an MQTT notification sequence. Claim 6 A voice-based AI healthcare method characterized by the following steps: collecting voice from a voice input / output unit; recognizing the collected voice in an ASR module and converting it into text; transmitting the recognition result by the ASR module to an SLM module to generate a response; synthesizing the response into voice through a TTS module and outputting it through a voice input / output unit; evaluating the user's health status by analyzing biological and speech information through a healthcare model; determining abnormal signs based on the evaluation result of the healthcare model in an abnormal signal detection module; transmitting a notification to an MQTT broker-based communication module when an abnormality is determined; and coordinating time-based events through a scheduling agent, wherein the scheduling agent checks the status flags of the abnormal signal detection module and the healthcare model at the time of triggering, requests the TTS module to generate a prompt, and simultaneously issues a notification through the MQTT broker-based communication module, and the ASR module constantly monitors urgent keywords and, when the detection reliability is above a threshold, pushes an event to the abnormal signal detection module regardless of the processing path of the SLM module to trigger an MQTT notification sequence. Claim 7 A voice-based AI healthcare method according to claim 6, wherein the voice-based healthcare method further includes the step of calculating a similarity with domain knowledge based on the recognition result of the ASR module in a vector search engine, and wherein the SLM module is configured to generate response candidates by combining the context recovered from the vector search engine with the real-time conversation context, and to automatically include user reminders or additional questions by comparing the response candidates with scheduled events of a schedule agent.

Images