Monitoring support system, monitored terminal, server, and program for guardian terminal

Abstract

To provide simple and appropriate responses to the person being monitored, depending on the situation. [Solution] The monitored terminal (100) transmits the acquired voice data and location information to the server (200). The server (200) converts the voice data into text and analyzes the acoustic features to generate a summary and speech intent (including urgency level), and sends notification data to the guardian terminal (300) that includes the summary, location information, and multiple response candidate codes (including action candidate codes) corresponding to the speech intent. The guardian terminal (300) displays the notification data as a push notification (500A), and displays the summary (503), map thumbnail (504), and response buttons (505A, 505B), etc., along with a display according to the urgency level (e.g., icon 501, color change). When the guardian selects a response button, the corresponding code is sent to the monitored terminal (100) via the server (200), and the monitored terminal (100) executes a predetermined output.

Description

Technical Field

[0001] The present disclosure relates to a monitoring support system including a monitored terminal, a server, and a guardian terminal.

Background Art

[0002] In recent years, in order to confirm the safety of monitored persons such as children and the elderly, monitoring terminals equipped with functions such as GPS (Global Positioning System) have become widespread. Many of these terminals have a function of notifying the current position of the monitored person to a guardian's smartphone or the like. In addition, some terminals have a function of transmitting and receiving voice messages, enabling communication between the monitored person and the guardian.

[0003] However, with the conventional voice message function, the guardian had to play the received voice message to check the content. Therefore, in situations where it is difficult to hear the voice, such as during a meeting, while moving, or while driving, it was impossible to immediately grasp the content of the message, and there was a problem that it was difficult to respond quickly.

[0004] In addition, when responding from the guardian to the monitored person, it was necessary to create a message by free input or select a fixed set phrase defined in advance, and there was a problem that it was cumbersome to quickly make an appropriate response according to the situation. Furthermore, many of the response options (buttons) displayed by push notifications or the like were static, and no optimization was performed according to the usage situation of the user. Also, Patent Document 1 describes a technique related to an automatic recommended response to a received message.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] One of the purposes of this disclosure is to provide appropriate responses to the person being monitored, depending on the situation. [Means for solving the problem]

[0007] To solve the above problems, a monitoring support system according to one aspect of the present disclosure includes a monitored terminal, a server, and a guardian terminal, wherein the monitored terminal transmits acquired voice data and current location information to the server, the server converts the voice data into text to generate a summary and utterance intent, and transmits notification data including a plurality of response candidate codes corresponding to the summary, location information, and utterance intent to the guardian terminal, the guardian terminal pushes the notification data and returns the selected response candidate code to the server, the server transmits response data including the received response candidate code to the monitored terminal, and the monitored terminal performs a predetermined output based on the response data. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide appropriate responses to the person being monitored, depending on the situation. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of a monitoring support system 1 according to one embodiment of the present disclosure. [Figure 2A] This is a block diagram showing the hardware configuration of a monitored terminal 100 according to one embodiment of the present disclosure. [Figure 2B] This is a block diagram showing the functional configuration of a monitored terminal 100 according to one embodiment of the present disclosure. [Figure 3A] This block diagram shows the hardware configuration of server 200 according to one embodiment of the present disclosure. [Figure 3B]This is a block diagram (extended version) showing the functional configuration of server 200 according to one embodiment of this disclosure. [Figure 4A] This is a block diagram showing the hardware configuration of a parental terminal 300 according to one embodiment of the present disclosure. [Figure 4B] This is a block diagram showing the functional configuration of a parental terminal 300 according to one embodiment of the present disclosure. [Figure 5A] This figure shows an example of a push notification 500A (first display state) displayed on a parent terminal 300 according to one embodiment of this disclosure. [Figure 5B] This figure shows an example of a push notification 500B (second display state) displayed on a parent terminal 300 according to one embodiment of this disclosure. [Figure 6] This is a sequence diagram showing the processing flow in a monitoring support system 1 according to one embodiment of the present disclosure. [Figure 7] This is a flowchart showing the processing flow in server 200 according to one embodiment of this disclosure. [Figure 8] This is a flowchart showing the flow of push notification display and response processing in a parent terminal 300 according to one embodiment of this disclosure. [Figure 9] This figure shows an example of the data structure of notification data 900 according to one embodiment of this disclosure. [Figure 10] This is a conceptual flowchart of an acoustic analysis process according to one embodiment of the present disclosure. [Figure 11] This is a conceptual flowchart of the response candidate generation process according to one embodiment of the present disclosure. [Figure 12A] This is an example of the UI display of a parent terminal 300 according to one embodiment of this disclosure (in an emergency). [Figure 12B] This is an example of the UI display (in-app notification / evidence presentation) of a parent terminal 300 according to one embodiment of this disclosure. [Figure 13] This is a schematic diagram of a distributed processing architecture relating to a modified version of the present disclosure, where (a) represents a monitored terminal processing type, (b) represents a guardian terminal processing type, and (c) represents a P2P type. [Figure 14]A table showing an output example of the monitored terminal 100 according to an embodiment of the present disclosure. [Figure 15] A flowchart showing a conceptual machine learning life cycle of history analysis and learning processing according to an embodiment of the present disclosure. [Figure 16] A conceptual flowchart of guardian situation estimation and notification optimization processing according to an embodiment of the present disclosure. [Mode for Carrying Out the Invention]

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each figure, the same or corresponding components are denoted by the same reference numerals, and redundant descriptions are omitted as appropriate.

[0011] [Definition of Terms] In this specification, the "utterance intention" refers to information extracted from voice data that indicates the situation, desires, or emotional state of the speaker, and includes at least level information (urgency level) indicating the urgency of the guardian's response to the utterance. It is distinguished from information that simply classifies the topic (subject matter) related to the content of the utterance. The urgency level can be expressed, for example, in three levels of "normal", "attention", "emergency", or more levels. Note that in this specification, "generating an utterance intention" does not necessarily require a step of explicitly generating and recording an independent data item or intermediate code named "utterance intention". When voice data or information derived therefrom (text data, acoustic feature quantities, etc.) is input, information processing reflecting the situation, desires, or emotional state of the speaker is performed, and if the processing result affects subsequent processing (e.g., generation of response candidate codes, generation of notification data), throughout the information processing process, it can be interpreted that an "utterance intention is generated" and utilized substantially. In particular, when the set of output response candidate codes changes systematically according to the content of the input voice information, the internal state or processing logic that causes the change may correspond to the generation and utilization of a substantial "utterance intention".

[0012] In this specification, a response candidate code is considered to "correspond to an utterance intent" if the generated utterance intent (including the urgency level) is one of the factors that determine or change at least one of the types, content, number, or order of the multiple response candidate codes presented. For example, this includes cases where (1) a specific response candidate code (e.g., an action candidate code) is presented only when a specific utterance intent is detected, (2) the order in which response candidate codes are presented changes depending on the utterance intent, or (3) the entire set of response candidate codes presented changes depending on the utterance intent. Even if other factors (e.g., parent's selection history, current time, location information) are involved in determining the response candidate code, the utterance intent is considered to "correspond" as long as it is used as one of the factors described above.

[0013] In this specification, “Response Candidate Code” refers to identifying information that indicates the type, content, or action of response selected by the guardian, and its specific data format is not limited. It may include any form of information that allows the system to identify the guardian's selection and reflect it in subsequent processing (e.g., sending to the monitored terminal, executing output), such as numbers, strings, flags, or commands that instruct the transmission of specific data (e.g., voice, text) or specific actions (e.g., making a phone call).

[0014] In this specification, "summary" refers to information that presents the main points of the transcribed audio data in a shorter format than the original text, or in a format that makes the important parts of the original text identifiable. This includes not only "extractive summaries" generated by extracting the main sentences and phrases from the original text, and "generative summaries" generated as new sentences while preserving the meaning of the original text, but also information that displays the entire original text while highlighting (e.g., bolding, coloring, highlighting) keywords and phrases related to the speaker's intent and importance, which may also be considered a form of "summary" from a functional standpoint that helps parents quickly grasp the content.

[0015] In this specification, "push notification" refers to a display in which a parental device proactively presents information to the user based on notification data sent from a server. This may include not only displays using standard push notification functions provided by the operating system (OS) (banners, notification centers, lock screen displays, etc.), but also any form of proactive information presentation that is triggered by receiving information from a server, such as modal windows, pop-ups, notification bars, or dedicated display areas that appear on the screen of a particular application when that application is active or running in the background.

[0016] [1. System Configuration (See Figure 1)] Figure 1 is a block diagram showing the overall configuration of a monitoring support system 1 according to one embodiment of the present disclosure. The monitoring support system 1 includes a monitoring terminal 100 held by the person being monitored, a guardian terminal 300 held by the guardian, and a server 200 connected to these via a communication network NW. (a) Name and code: Monitoring support system 1 (b) Specific implementation example: The system is constructed with the monitored terminal 100, the server 200, and the guardian terminal 300 working together via a communication network NW. Each device can be implemented as dedicated hardware or as software running on a general-purpose computer. (c) Core function: The server 200 processes the voice data and location information acquired by the monitored terminal 100, notifies the guardian terminal 300 of the results (summary, intended speech, response candidates), and feeds back the guardian's response to the monitored terminal 100, thereby supporting the understanding of the situation of the monitored person in a remote location and communication. (f) Coordination with other elements: The monitored terminal 100, the server 200, and the guardian terminal 300 send and receive information from each other via the communication network NW.

[0017] The monitored terminal 100 can be implemented as a small wearable device carried by the person being monitored, such as a child or the elderly, a dedicated terminal, or an application implemented on a general-purpose device such as a smartphone. Its main functions are to acquire the voice and location information of the person being monitored and transmit it to the server 200, and to execute predetermined outputs based on the response data from the server 200.

[0018] Server 200 can be implemented as one or more physical servers or as a virtual server built on a cloud computing environment. Its main functions are to receive information from the monitored terminal 100, perform advanced information processing (speech-to-text conversion, summarization, intent generation, response candidate generation, etc.), notify the guardian terminal 300 of the results, and relay the response from the guardian terminal 300 to the monitored terminal 100.

[0019] The parent terminal 300 can be implemented as a dedicated application installed on a general-purpose device such as a smartphone, tablet, or PC, or as a web application that runs on a web browser. Its main functions are to receive and display notification data from the server 200, and to accept and send responses from the parent to the server 200.

[0020] Communication networks (NW) can utilize various communication standards and technologies, such as the Internet, mobile phone networks (LTE, 5G, etc.), Wi-Fi, Bluetooth®, and LPWA (Low Power Wide Area) networks, either individually or in combination.

[0021] [2. Hardware Configuration] Next, we will describe the hardware configuration of each device. Monitoring terminal 100 (see Figure 2A) Figure 2A is a block diagram showing the hardware configuration of the monitored terminal 100. (a) Name and code: Monitoring terminal 100 (b) Specific implementation example: A wearable device in which each component is built into a dedicated small enclosure. (c) Core functions: Voice input, location information positioning, data transmission and reception, and user feedback output. (d) Internal structure and stored data: It comprises a control unit 101, a storage unit 102, a communication unit 103, a microphone 104, a GPS receiver 105, an output unit 106, and an operation unit 107. The storage unit 102 stores the OS, control program, temporary voice data and location data, etc. (e) Specific processing steps: The control unit 101 acquires voice using the microphone 104 and location information using the GPS receiver 105, based on instructions from the operation unit 107 or an automatic trigger. This information is transmitted to the server 200 via the communication unit 103. Based on the response data received from the server 200, the control unit 101 controls the output unit 106 (speaker, vibrator, LED, etc.) to output sound, vibration, light, etc. (f) Coordination with other elements: The communication unit 103 communicates with the server 200. The microphone 104, GPS receiver 105, and operation unit 107 are connected to the control unit 101. The control unit 101 controls the output unit 106.

[0022] The control unit 101 is a CPU (Central Processing Unit), MPU (Micro Processing Unit), SoC (System on a Chip), etc., and comprehensively controls the operation of the entire monitored terminal 100 by executing programs stored in the memory unit 102.

[0023] The memory unit 102 includes RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc., and stores the OS, control programs, applications, audio data, location information, setting information, etc.

[0024] The communication unit 103 is equipped with a cellular communication module (LTE, 5G, etc.), a Wi-Fi module, a Bluetooth module, an LPWA module, etc., and communicates data with the server 200 via the communication network NW.

[0025] Microphone 104 acquires the speech of the person being monitored and surrounding sounds as analog signals and converts them into digital audio data via an A / D converter (not shown).

[0026] The GPS receiver 105 receives signals from GPS satellites to determine the current location information (latitude, longitude, altitude, etc.). In addition to GPS, auxiliary positioning methods (A-GPS, Wi-Fi positioning, etc.) using information from Wi-Fi access points and cell phone base stations may also be used.

[0027] The output unit 106 may include multiple output devices for presenting information to the user, such as a speaker, a vibrator, and an LED (Light Emitting Diode). The speaker plays voice messages or warning sounds, the vibrator provides notifications through vibration, and the LED indicates the status with lighting and flashing patterns.

[0028] The control unit 107 includes buttons and switches for receiving physical input from the user, such as a power button, a recording start / stop button, and an SOS button. If a touchscreen is integrated with the output unit 106, touch operation may also function as part of the control unit 107.

[0029] Server 200 (see Figure 3A) Figure 3A is a block diagram showing the hardware configuration of server 200. (a) Name and code: Server 200 (b) Specific implementation examples: Rack-mount servers installed in a data center, or virtual server clusters built on cloud platforms such as AWS, Google Cloud, or Azure. (c) Core function: It processes access from multiple monitored terminals 100 and guardian terminals 300, and performs core information processing of the system, such as voice data analysis, intent estimation, notification generation, response relay, and data storage and learning. (d) Internal structure and stored data: The system comprises a control unit 201, a storage unit 202, and a communication unit 203. The storage unit 202 stores the OS, web server program, application server program, database management system (DBMS), machine learning model, user account information, history data, etc. (e) Specific processing steps: The control unit 201 receives voice data and location information from the monitored terminal 100 via the communication unit 203, and performs voice processing, notification generation, etc., using programs and models stored in the memory unit 202. The generated notification data is sent to the guardian terminal 300, and the response from the guardian terminal 300 is relayed to the monitored terminal 100. For details, please refer to the description of the functional blocks described later and the flowchart in Figure 7. (f) Coordination with other elements: The communication unit 203 connects with the monitored terminal 100 and the guardian terminal 300 via the communication network NW.

[0030] The control unit 201 consists of one or more high-performance CPUs, GPUs (Graphics Processing Units, especially for machine learning processing), TPUs (Tensor Processing Units), etc., and executes programs stored in the memory unit 202 to perform advanced processing for the entire server 200.

[0031] The memory unit 202 combines a large-capacity RAM, a high-speed SSD (Solid State Drive), an HDD (Hard Disk Drive), etc., and permanently or temporarily stores the OS, middleware, application programs, and various data (user information, voice data, text data, location information, speech intent data, response candidate data, machine learning models, log data, etc.).

[0032] The communication unit 203 is equipped with a high-speed network interface (e.g., Ethernet®) and performs simultaneous data communication with numerous terminals via the communication network NW. Load balancers and firewalls (not shown) may also operate in conjunction with it.

[0033] Parental device 300 (see Figure 4A) Figure 4A is a block diagram showing the hardware configuration of the parent terminal 300. (a) Name and code: Parent terminal 300 (b) Specific examples of implementation: Smartphones, tablet devices, PCs, etc. (c) Core function: Receive and display notifications from server 200, accept response operations from parents, and send them to server 200. (d) Internal structure and stored data: It comprises a control unit 301, a storage unit 302, a communication unit 303, a display unit 304, an operation unit 305, and a speaker 306. The storage unit 302 stores the OS, a dedicated monitoring application, notification history, setting information, etc. (e) Specific processing steps: The control unit 301 receives notification data from the server 200 via the communication unit 303 and displays it on the display unit 304 as a push notification or in-app notification. It accepts a response selection from the operation unit 305 (touch panel, etc.) and sends the corresponding response candidate code to the server 200 via the communication unit 303. For details, please refer to the description of the functional blocks described later and the flowchart in Figure 8. (f) Coordination with other elements: The communication unit 303 is connected to the server 200 via the communication network NW. The display unit 304, the operation unit 305, and the speaker 306 are connected to the control unit 301.

[0034] The control unit 301 is a CPU or SoC installed in a smartphone or the like, and controls the overall operation of the parent terminal 300 by executing the OS and application programs stored in the memory unit 302.

[0035] The memory unit 302 includes RAM, ROM, flash memory, etc., and stores the OS, a dedicated monitoring application, notification data, configuration information, cache data, and the like.

[0036] The communication unit 303 is equipped with a cellular communication module, a Wi-Fi module, a Bluetooth module, etc., and communicates data with the server 200 via the communication network NW.

[0037] The display unit 304 is a touch panel display such as a liquid crystal display or an organic EL display, and displays push notifications, application UI screens, map information, message content, etc.

[0038] The control unit 305 is a touch panel integrated with the display unit 304, or physical buttons (volume buttons, power button, etc.), and accepts input from parents such as taps, swipes, and long presses.

[0039] Speaker 306 plays notification sounds, alert sounds, or received voice messages (in the case of free response).

[0040] [3. Functional Block Configuration] Next, we will describe the functional block configuration of each device. These functional blocks are primarily realized by the control unit of each device executing programs stored in its memory unit. Monitoring terminal 100 (see Figure 2B) Figure 2B is a block diagram showing the functional configuration of the monitored terminal 100. (a) Names and codes: Voice acquisition unit 111, location information acquisition unit 112, transmission unit 113, reception unit 114, output control unit 115 (b) Specific implementation example: A group of software modules executed by the control unit 101. (c) Core functions: Acquisition and transmission of voice and location information, reception of responses from the server and output control. (d) Internal structure and stored data: Each part operates in coordination. The transmitting unit 113 may temporarily use the transmitting buffer, and the receiving unit 114 may temporarily use the receiving buffer. The output control unit 115 may store or refer to setting data (read from the storage unit 102 or received from the server) that defines the correspondence between response codes and output content (audio files, vibration patterns, LED patterns).

[0041] (e) Specific processing steps: Voice acquisition unit 111: Acquires the voice of the person being monitored via microphone 104 and generates digital voice data (e.g., PCM, AAC format). Noise cancellation processing and voice enhancement processing may be performed.

[0042] Location information acquisition unit 112: Acquires current location information (latitude, longitude, timestamp, accuracy information, etc.) using the GPS receiver unit 105 or other positioning means.

[0043] Transmitting unit 113: Transmits the voice data acquired by the voice acquisition unit 111 and the current location information acquired by the location information acquisition unit 112, along with metadata such as the person being monitored and timestamp, to the server 200 via the communication unit 103. It is desirable that the data be transmitted encrypted. This corresponds to the "transmitting unit" in Appendix 2.

[0044] Receiving unit 114: Receives response data, including the response candidate code selected by the parent terminal 300, from the server 200 via the communication unit 103. This corresponds to the "receiving unit" in Appendix 2.

[0045] Output control unit 115: Based on the response candidate code contained in the response data received by the receiving unit 114, it causes the output unit 106 (speaker, vibrator, LED, etc.) to execute a predetermined output. This output can be classified into the following intermediate concepts depending on its purpose (see Figure 14 for details). (1) Response transmission output: An output that conveys feedback from the guardian to the person being monitored by playing a response message from the guardian (e.g., audio data or text-to-speech instructions received from the server) as audio or by vibrating briefly in a specific pattern. (2) Warning output: An output used by a guardian to alert the person being monitored (e.g., a ringing tone in response to "Call me", a specific LED flashing). (3) Emergency alarm output: An output to alert the person being monitored and those around them to danger when the guardian determines that an emergency has occurred or when the server detects an emergency (e.g., a loud buzzer, strong vibrations, rapid flashing of LEDs). The output control unit 115 appropriately controls the type, intensity, duration, etc. of these outputs based on the information contained in the response candidate code (e.g., output type, message content, intensity level). This corresponds to the "output unit" in Appendix 2.

[0046] (f) Coordination with other elements: The voice acquisition unit 111 cooperates with the microphone 104, and the location information acquisition unit 112 cooperates with the GPS receiver 105. The transmission unit 113 and the receiver 114 cooperate with the communication unit 103. The output control unit 115 cooperates with the output unit 106.

[0047] Server 200 (See Figures 3B, 10, 11, 15, and 16) Figure 3B is an extended block diagram showing the functional configuration of server 200. Server 200 functions as a receiving unit 211, voice processing unit 212, notification generation unit 213, transmission unit 214, and response processing unit 215, with the control unit 201 executing the program in the storage unit 202. Furthermore, it may include a history analysis / learning processing unit 216, a parent status estimation unit 217, and a notification optimization processing unit 218.

[0048] (a) Names and codes: Receiving unit 211, voice processing unit 212, notification generation unit 213, transmission unit 214, response processing unit 215, history analysis / learning processing unit 216, parent status estimation unit 217, notification optimization processing unit 218 (b) Specific implementation example: A group of software modules executed by the control unit 201. These may be implemented as a microservices architecture, with each function being able to scale independently. (c) Core function: Information processing and control of the entire system. (d) Internal structure and stored data: Details of each functional unit will be described later. Overall, the user account DB, terminal management DB, message history DB, learning model repository, configuration information DB, etc. are stored and managed in the storage unit 202.

[0049] (e) Specific processing steps: As shown in the flowchart in Figure 7, a series of processes are executed from receiving data from the monitored terminal 100 to sending a notification to the guardian terminal 300 and processing the response. History analysis and learning processes, guardian status estimation, and notification optimization are executed in conjunction with these basic processes or asynchronously.

[0050] (f) Coordination with other elements: The receiving unit 211 and the transmitting unit 214 communicate with an external terminal via the communication unit 203. Each internal functional block exchanges data with each other and cooperates.

[0051] The receiving unit 211 receives data including voice data and location information from the monitored terminal 100 via the communication unit 203, and also receives data including a candidate response code from the guardian terminal 300. After the received data is authenticated and verified, it is passed to the appropriate processing unit (e.g., voice processing unit 212, response processing unit 215).

[0052] The voice processing unit 212 (see Figure 3B, S702-S704 in Figure 7, and Figure 10) analyzes the voice data received by the receiving unit 211 and generates information (text data, summary sentences, and speech intent) necessary for the guardian to understand the situation of the person being monitored and make appropriate decisions. Internally, it has a voice-to-text unit 212a, a summary sentence generation unit 212b, and a speech intent generation unit 212c (including a text analysis unit 212d and an acoustic analysis unit 212e).

[0053] The speech-to-text unit 212a converts the received speech data into text data. The summary generation unit 212b creates a summary from the generated text data. Possible types of summaries (intermediate concepts) include information reduction summaries, important information extraction summaries, keyword presentation summaries, and sentiment conveying summaries. The speech intent generation unit 212c generates multifaceted speech intents based on text data and acoustic features. The text analysis unit 212d analyzes the text data and estimates the subject, request content, emotional polarity, etc. The acoustic analysis unit 212e (see Figure 10) extracts acoustic features from the audio data (Figure 10, "Acoustic Feature Extraction") and uses a machine learning model (Figure 10, "Machine Learning Model") to determine the urgency level and emotion category. This model is pre-trained using a large number of audio samples and labels. The speech intent generation unit 212c integrates these analysis results to generate speech intent (urgency level, emotion category information, action request information, situation context information, etc.).

[0054] The notification generation unit 213 (see Figure 3B, S705-S706 in Figure 7, and Figure 11) integrates information from the voice processing unit 212, location information, and information from other processing units to generate notification data (see Figure 9) for transmission to the parent terminal 300. In particular, it generates a list of optimal response candidate codes according to the speaker's intent and the situation. The logic for generating the response candidate code list (see Figure 11) takes utterance intent, context information, and history information as input and appropriately combines candidates of intermediate concepts (intent-driven, context-dependent, learning-based, action-executing, and free-response types). Notification data includes a summary, information indicating the speaker's intent, location information, and a list of suggested response codes.

[0055] The transmitting unit 214 transmits the notification data generated by the notification generation unit 213 to the appropriate parent terminal 300 via the communication unit 203. It also transmits response data to the monitored terminal 100 based on instructions from the response processing unit 215.

[0056] The response processing unit 215 interprets the candidate response code received by the receiving unit 211 from the guardian terminal 300, generates the corresponding response data, and instructs the transmission unit 214 to send it to the monitored terminal 100. It also records the response content in the history analysis and learning processing unit 216.

[0057] The history analysis and learning processing unit 216 (see Figures 3B and 15) stores and analyzes various history data collected throughout the system, continuously personalizing or generalizing the model of the voice processing unit 212 and the logic of the notification generation unit 213 to improve system performance. It also detects changes and anomalies in the behavioral patterns of the person being monitored. Figure 15 shows the conceptual machine learning lifecycle in this processing unit, which includes steps such as data collection, preprocessing, feature engineering, model learning and evaluation, and model deployment.

[0058] The parent status estimation unit 217 (see Figures 3B and 16) estimates whether the parent is currently available to respond and what kind of response would be desirable (e.g., driving, in a meeting, traveling, on a break) based on information obtained from the parent terminal 300 (with explicit consent) (e.g., calendar information, location information, terminal usage status, accelerometer information, ambient sound level from microphone input, etc.). Figure 16 shows the conceptual link between this estimation process and the notification optimization process described later.

[0059] The notification optimization processing unit 218 (see Figures 3B and 16) takes as input the parent status estimated by the parent status estimation unit 217, the urgency of the message determined by the voice processing unit 212, the communication environment, etc. (Figure 16 "Input Information"), and dynamically determines the notification method (push, in-app, SMS, etc.), notification volume, amount of information to display (summary only, full text + summary, etc.), and the number and types of response candidates to present, based on pre-set rules and learning models (Figure 16 "Notification Optimization Rule Engine"). The determined notification method and content are transmitted to the notification generation unit 213 and reflected in the actual notification data generation (Figure 16 "Optimized Notification Parameters").

[0060] Parental device 300 (see Figures 4B, 5AB, 8, and 12AB) Figure 4B is a block diagram showing the functional configuration of the parent terminal 300. (a) Names and symbols: Receiving unit 311, display control unit 312, operation reception unit 313, transmitting unit 314 (b) Specific implementation example: A set of software modules for a dedicated monitoring application executed by the control unit 301. (c) Core functions: Receiving and displaying notifications from the server, receiving and sending responses from parents. (d) Internal structure and stored data: The display control unit 312 stores UI templates and display rules. The operation reception unit 313 processes touch events, etc. (e) Specific processing steps (see Figure 8): Receiving unit 311: Receives notification data (see Figure 9) from server 200 (S801). Display control unit 312: Based on the received notification data, it generates and displays a push notification (Figures 5A, 5B) or an in-app notification (Figure 12B) (S802, S803). The displayed content changes dynamically according to the urgency level in the notification data (e.g., in an emergency, a warning is displayed as in Figure 12A). The response candidate buttons may also display the reasoning behind their suggestion (Figure 12B, modified example).

[0061] reference). Operation reception unit 313: Accepts responses from parents to select response buttons (tap, long tap, etc.) and to select notification units (S804-S807, S810). Transmitter 314: Sends a response candidate code corresponding to the selected response candidate to the server 200 (S809). (f) Coordination with other elements: The receiving unit 311 and the transmitting unit 314 cooperate with the communication unit 303. The display control unit 312 cooperates with the display unit 304. The operation reception unit 313 cooperates with the operation unit 305 (touch panel, etc.).

[0062] [4. Push notification UI (See Figures 5A, 5B, 12A, and 12B)] Figures 5A and 5B show examples of push notifications displayed on the parent device 300. Figure 12A shows an example of display in an emergency, and Figure 12B shows an example of display including in-app notifications and justification for response options. Figure 5A shows a push notification 500A in its initial display state (e.g., a normal banner notification). The push notification 500A displays an icon 501 indicating the intent of the speech, the name and icon 502 of the person being monitored, a summary text 503, a map thumbnail image 504 indicating location information, and multiple response buttons (e.g., response button 505A, response button 505B).

[0063] The display control unit 312 changes the display mode of the push notification 500A / 500B according to the speech intent code 908 included in the received notification data, particularly the associated urgency level. For example, it changes the icon 501 indicating the speech intent according to the urgency level (e.g., a smile icon in normal times, and a warning mark icon as shown in Figure 12A in emergencies), and furthermore, in the case of an "urgent" level, the entire notification (background color, border, etc.) is displayed in red (corresponding to appendices 5 and 9). This allows parents to instantly and intuitively grasp the urgency of a message with just a glance at the push notification.

[0064] The map thumbnail image 504 shows the location information of the monitored device 100 and is displayed together with the summary text 503 (corresponding to Appendix 6). This allows the guardian to understand the location of the monitored person at the same time as the content of the message.

[0065] The response buttons 505A and 505B correspond to response candidate codes sent from the server 200. These may include the aforementioned action candidate codes in addition to the normal response message transmission codes. When a parent taps a button corresponding to an action candidate code (e.g., the "Call Now" button, the "Check on Map" button), the parent terminal 300 can send a response code to the server 200 and, at the same time, or based on instructions from the server 200, automatically execute the corresponding action (making a phone call, launching a map application, etc.).

[0066] The server 200 or parent terminal 300 can learn the past selection history of response buttons on the parent terminal 300 and arrange the response buttons in order of their estimated selection rate (corresponding to Appendix 7). For example, if it is determined to be an "urgent" situation and the time is evening, and it has been learned that the "Pick up (route guidance)" button has a high selection rate, then that button will be placed in the most easily accessible position. In addition, based on the urgency level and learning results, the response buttons that are considered most likely to be selected (especially important action candidate buttons) will be placed in the most easily accessible positions (e.g., far right, top, etc.).

[0067] When the parent terminal 300 receives a long tap (long press) operation on the push notification 500A, it can switch to the second display state of the push notification 500B as shown in Figure 5B (corresponding to Appendix 8). In the second display state 500B, more response buttons (e.g., response buttons 505A to 505D) are displayed than in the first display state 500A. This allows for a simple display under normal circumstances, while providing more options (additional response buttons including a variety of action candidates) when necessary.

[0068] Push notifications 500A or 500B can display a countdown bar 506 (corresponding to Appendix 10). This indicates that if a predetermined time elapses without the response button being selected, a pre-configured default response (e.g., "Checking") is automatically sent to the server 200. This ensures that even if the guardian cannot respond immediately, some form of feedback is automatically provided to the person being monitored, reducing their anxiety.

[0069] When a parent selects (tap) the push notification 500A or 500B itself (parts other than the button), the parent's device 300 can transition to a dedicated application screen (corresponding to Appendix 11). On this application screen, it is possible to play the original voice message or display a more detailed map.

[0070] When any response button (e.g., 505A) is selected, the parent terminal 300 can update the push notification 500A / 500B, or the selected response button 505A, to indicate that it has been responded to (e.g., grayed out, with a checkmark) (corresponding to Appendix 12). This makes it easy to determine which notification has been responded to.

[0071] [5. Processing Flow (See Figures 6, 7, and 8)] Figure 6 is a sequence diagram showing the processing flow in the monitoring support system 1. First, voice is acquired (S601) and location information is acquired (S602) at the monitored terminal 100. The monitored terminal 100 transmits the acquired voice data and location information to the server 200 (S603).

[0072] Server 200 converts the received audio data into text and generates a summary and the speaker's intent (including the urgency level) (S604). Then, it generates notification data including the summary, location information, the speaker's intent, and a response candidate code including an action candidate code (S605), and sends it to the parent terminal 300 (S606).

[0073] The parent terminal 300 receives notification data and displays a push notification (S607). At this time, display control is performed according to the urgency level. When the parent selects a response button (S608), the corresponding response candidate code is sent back to the server 200 (S609). If the selected button is an action candidate button, the corresponding action may be executed. Server 200 receives a candidate response code, generates response data including it (S610), and sends it to the monitored terminal 100 (S611).

[0074] The monitored terminal 100 receives the response data and performs a predetermined output (sound, vibration, light, etc.) corresponding to the code (S612).

[0075] Figure 7 is a flowchart showing the details of the processing (S604, S605) in server 200. When server 200 receives voice data and location information from the monitored terminal 100 (S701), it converts the voice data into text using STT processing (S702). Next, it analyzes the text data and generates a summary (S703). Simultaneously, it analyzes the acoustic features of the text data and voice data to classify and generate the utterance intent and urgency level (S704). Then, it selects or generates multiple response candidate codes (including action candidate codes) corresponding to the generated utterance intent (urgency level) (S705). Finally, it generates notification data including the summary, location information, utterance intent, and response candidate codes (S706).

[0076] Figure 8 is a flowchart detailing the processing (S607, S608, S609) on the parent terminal 300. When the parent terminal 300 receives notification data from the server 200 (S801), it generates a push notification (S802). At this time, it controls the display considering the speech intent icon (Note 5) according to the urgency level, the map thumbnail (Note 6), the order of response buttons (Note 7), the abnormal notification display (Note 9), the countdown bar (Note 10), etc. Then, it displays the push notification (S803).

[0077] Next, the system waits for user input (S804). If a long tap operation is performed (S805:YES), the display state is switched to the second display state, and the number of buttons is increased (S806, Note 8). If a response button is selected (S807:YES), the display is updated to "responded" (S808, Note 12), and the selected response candidate code is sent to the server 200 (S809). If necessary, the process corresponding to the selected action candidate is executed. If a notification body is selected (S810:YES), the system transitions to the application screen (S811, Note 11).

[0078] [6. Data Structure (See Figure 9)] Figure 9 shows an example of the data structure of notification data 900 generated by server 200 and sent to parent terminal 300. The notification data 900 may include message ID 901, timestamp 902, person being monitored ID 903, location information 904, voice data (or a reference thereof) 905, text data 906, summary text 907, speech intent code 908 (including urgency level information, emotion category information, action request information, situation context information, etc.), response candidate code list 909 (including response candidate codes of various intermediate concepts such as intent-based, context-dependent, learning-based, action-executing, and free-response types), and an anomaly classification flag 910. The parent terminal 300 uses this information to display a push notification as shown in Figures 5A, 5B, 12A, and 12B.

[0079] [7. Variant] This disclosure is not limited to the embodiments described above, and various modifications are possible.

[0080] (Modified version: Emergency-linked notification sound) In the above embodiment, the UI display pattern was changed according to the urgency level. In addition to this, or instead, the parent terminal 300 may be configured to change the notification sound or vibration pattern of the push notification according to the urgency level included in the received notification data. For example, a normal notification sound could be used for the "normal" level, a slightly longer notification sound for the "caution" level, and a loud, repeated warning sound or a special vibration pattern for the "urgent" level. This allows the parent to understand the urgency level even when not looking at the terminal screen, and furthermore, it can be shown that the urgency level generated by the server is being used in a way that is observable from the outside, either audibly or tactilely.

[0081] (Variation: Providing justification for candidate responses) The display control unit 312 of the parent terminal 300 may, when displaying each response button 505A, etc. in a push notification 500A / 500B (or an in-app notification as shown in Figure 12B), additional information indicating the main reason why that response button was presented may be displayed. For example, it may display text such as "SOS detected" or an icon such as "Recommended from history". This information can be realized by the server 200 adding rationale information (e.g., flags indicating intent-based, history-based, action candidate, etc.) to each response candidate code when generating notification data. This configuration not only makes it easier for parents to understand why that button is presented, but also allows them to objectively confirm from the parent terminal's display screen that their utterance intent actually corresponds to (involves) the generation of the response candidate.

[0082] (Modified configuration: Distributed processing configuration) In the above embodiment, a configuration was described in which the server 200 performs major speech processing (text conversion, summarization, intent generation) and notification generation in a unified manner, but this disclosure is not limited thereto. Figure 13 shows a schematic diagram of a distributed processing architecture according to this modified example. For example, the following distributed processing configuration is also possible.

[0083] (a) Main processing on the monitored terminal (see Figure 13(a)): The monitored terminal 100 uses its built-in processing unit (e.g., AI chip) to perform all or most of the transcription of voice data into text, generation of a summary, and generation of speech intent, and transmits notification data including the results to the guardian terminal 300 via the server 200 (which in this case mainly functions as a relay / notification hub) or directly via P2P communication, etc.

[0084] (b) Main processing on the parent terminal (see Figure 13(b)): Although the server 200 receives the voice data and location information, it only performs text conversion and basic intent determination. The parent terminal 300 performs summary generation, detailed intent determination, and response candidate code generation after receiving the data.

[0085] (c) P2P communication (see Figure 13(c)): Without going through the server 200 at all, the monitored terminal 100 and the guardian terminal 300 communicate directly using Wi-Fi Direct, Bluetooth, or other P2P technology and execute the same processing flow as in (a) or (b) above.

[0086] Even with these distributed processing configurations, as long as the basic technical concept and information processing flow of this disclosure are realized—that based on voice data acquired or received by one device (a monitored terminal, a server, or a guardian terminal), another device (a guardian terminal) can understand the content and intent of the speech, and a response candidate corresponding to the intent is presented, enabling a quick response—then even if some or all of the role of the "server" in Appendix 1 is performed by another device (a monitored terminal, a guardian terminal), it may be included in the scope of this disclosure or should be considered equivalent.

[0087] (Variation: Detailed logging and auditing functions) To ensure the transparency and reliability of its operation, it is desirable that Server 200 log key data and their timestamps in a series of processes, such as received audio data (or its features), generated text, summary text, determined utterance intent and urgency level, generated response candidate code list, response code received from the parent terminal, and response data sent to the monitored terminal, associated with each message ID. This log can be used not only for debugging and improving the system's performance, but also as an audit trail to retrospectively verify whether the service provided meets the constituent requirements of this disclosure. For example, it becomes possible to statistically analyze the accuracy of urgency determination over a specific period, and the correspondence between utterance intent and response candidates.

[0088] (Example: End-to-End type processing) The speech processing unit 212 of the server 200 may be configured using one or more end-to-end machine learning models that take speech data or derived information as input and directly output a summary sentence and response candidate codes, instead of performing transcription, summarization, and intent generation in separate modules. Such models are designed to internally represent and process the speaker's intent and urgency contained in the input information through the learning process. Therefore, even if independent “speech intent” data is not explicitly generated in the system architecture, as long as the model’s output (response candidate codes) changes to reflect the intent of the input speech, it can be considered functionally “generating speech intent” and “generating response candidate codes that correspond to speech intent,” and thus may be included in the technical ideas of this disclosure.

[0089] (Variation: Free response function) The response options presented to the parent terminal 300 may include not only standard responses but also options for the parent to record a free voice message or input a text message. If the parent selects one of these options, the parent terminal 300 sends the recorded voice data or the entered text data to the server 200. In this case, the voice data or text data itself, or the information instructing them to be sent, effectively functions as a "response candidate code" and is sent to the server 200. The server 200 sends the received voice data or text data to the monitored terminal 100. The monitored terminal 100 performs a "predetermined output" by playing the received voice data or displaying or reading aloud the text data. Even in such a free response format, the technical idea of ​​this disclosure can be included in the fact that the parent selects a response from the options presented according to their intent to speak, and this is conveyed to the monitored terminal.

[0090] (Variation: Keyword highlighting) Instead of generating a summary, or in addition to generating a summary, the voice processing unit 212 of the server 200 may analyze the entire transcribed data to identify keywords and phrases related to the speaker's intent and generate data including their location information and emphasis indications. Based on this information, the parent terminal 300 highlights specific keywords, etc., when displaying the full text. This configuration, too, can be included in the technical concept of this disclosure as it helps parents quickly understand the content.

[0091] (Variation: In-app notification display) When the parent terminal 300 receives notification data from the server 200, instead of using the OS's standard push notification function, or in addition to that, it may display the notification content (summary, icon, map, response button, etc.) directly within a dedicated monitoring application (see Figure 12B). For example, a notification banner may be displayed at the top of the application screen, a dedicated notification list screen may be updated, or it may be displayed in a modal window at the forefront. In this case as well, the fact that the parent can understand the content and respond based on the notification data from the server remains the same and can be included in the technical concept of this disclosure.

[0092] (Modified version: Advanced technology through AI integration) (1) LLM utilization: The server 200 or the parent terminal 300 may utilize a Large-Scale Language Model (LLM). For example, based on the summary sentence and utterance intent generated by the speech processing unit 212, the LLM generates a more natural and detailed situational description for the parent. The notification generation unit 213 also uses the LLM to generate multiple diverse response candidate messages (not just codes, but specific reply drafts) that are more contextually relevant based on the utterance intent and the parent's past dialogue history, allowing the parent to select or edit them. Furthermore, on the monitored terminal 100 side, the LLM may function as a primary response agent that interactively confirms the situation with the monitored person based on instructions from the parent (e.g., "Listen in more detail").

[0093] (2) Multimodal emotion analysis: If the monitored terminal 100 is equipped with a camera or wearable sensors (described later), it transmits multimodal information such as voice, text, facial expressions, gaze, gestures, heart rate, and skin electrical activity to the server 200, and the voice processing unit 212 analyzes this information in an integrated manner to perform more accurate emotion estimation and understanding of mental state.

[0094] (3) Abnormal behavior detection AI: The history analysis and learning processing unit 216 learns the normal behavior patterns of the person being monitored (e.g., travel routes, activity times, tendencies in speech content, social relationships, etc.) and detects statistically significant deviations from there (e.g., going out late at night, frequent visits to unusual places, a sudden increase in negative speech, an increase / decrease in contact with specific people) as abnormal behavior, and automatically issues a warning notification to the guardian.

[0095] (Modified example: IoT / sensor integration) (1) Wearable device integration: The monitored terminal 100 integrates with wearable devices such as smartwatches and activity trackers to acquire biometric and activity information such as heart rate, blood oxygen saturation, sleep patterns, and fall detection. This information is transmitted to the server 200 along with voice data and used by the voice processing unit 212 as auxiliary information when determining the intent of the speech (especially the degree of urgency and health status). For example, if the heart rate is abnormally high even if the content of the speech is normal, the degree of urgency will be determined to be higher.

[0096] (2) Smart home device integration: When the person being monitored is at home, the system integrates with smart home devices such as smart speakers, smart lighting, smart locks, and indoor cameras to understand the person's activities inside the house (e.g., which room they are in, how they are using appliances, whether they have visitors), and integrates this information with the monitoring information to provide to the guardian.

[0097] (Variation: Application of privacy protection technology) This system may apply the following technologies as appropriate to enhance user privacy protection. (1) Federated Learning: When the history analysis and learning processing unit 216 learns and updates the machine learning model, it collects and integrates only the parameters of the model learned locally on each terminal, without aggregating each user's local data on the server.

[0098] (2) Differential Privacy: When publishing and sharing statistical information and trained models, appropriately designed noise is added so that individual user data cannot be identified.

[0099] (3) On-device AI processing: To the extent permitted by the processing capacity of the monitored terminal 100 or the guardian terminal 300, processes such as voice recognition, intent estimation, and response candidate generation are completed within the terminal, minimizing the transmission of personal information to the server.

[0100] (4) Data encryption and access control: Data in the communication path and in the storage unit 202 of the server 200 is always encrypted, and access rights are strictly controlled.

[0101] (Variation: Using blockchain) Blockchain technology may be used to record and manage important monitoring logs, particularly urgent notifications, parental responses, and related location information and timestamps, in order to ensure their authenticity, completeness, and non-tampering. This would enhance the evidentiary value in the event of an incident.

[0102] (Modification: Expansion of application fields) The technical concept presented in this disclosure is not limited to child monitoring but can be applied to a variety of other fields, such as those listed below. (a) Elderly care: Monitor the speech and sounds of elderly people living alone or in care facilities to detect abnormal health conditions, falls, emergencies, etc., and notify family members and care staff. This can also be applied to support communication with dementia patients.

[0103] (b) Telemedicine and healthcare: Analyzes the patient's voice, such as words related to symptoms or coughing, from their home to communicate the situation to doctors and nurses, supporting remote diagnosis and advice.

[0104] (c) Safety management at the workplace: Collect voice and biometric information from wearable devices worn by workers in factories, construction sites, mines, etc., to detect dangerous situations or health problems and warn managers and other workers.

[0105] (d) Pet monitoring: Analyzes pet noises and sounds while the owner is away and notifies the owner of any abnormal situations (illness, stress, intruder, etc.). Emotions and needs can be estimated from the sounds and used as triggers to remotely operate automatic pet feeders or air conditioners.

[0106] (e) Education and learning support: Analyze student speech (questions, discussions, presentations, etc.) and provide feedback to teachers and tutors on comprehension levels and areas of difficulty. It can also be used to revitalize communication in online learning environments.

[0107] Each embodiment and modification described herein may be combined, omitted, or replaced with other elements or steps as appropriate, without departing from the spirit of the invention.

[0108] [Note] [General tasks] It is difficult for guardians to quickly and easily understand the content of voice messages sent by the person being monitored without listening to the audio, and to easily provide an appropriate response depending on the situation.

[0109] [Note 1] Includes the monitored device, server, and guardian device, The monitored terminal comprises a transmitting unit that transmits acquired voice data and current location information to the server, a receiving unit that receives response data from the server including a candidate response code selected by the guardian terminal, and an output unit that performs a predetermined output based on the response data. The server comprises: an audio processing unit that converts the audio data received from the monitored terminal into text to generate a summary and a speech intent; a notification transmission unit that transmits notification data including a plurality of response candidate codes corresponding to the speech intent, the summary, and the location information of the monitored terminal to the guardian terminal; and a response transmission unit that transmits the response candidate codes received from the guardian terminal to the monitored terminal. The parent terminal includes a display control unit that receives the notification data from the server and displays a push notification including the summary text and the multiple response candidate codes, and a reply unit that sends back the selected response candidate code to the server. A monitoring and support system. (Issues corresponding to Appendix 1) It is difficult for guardians to quickly and easily understand the content of voice messages sent by the person being monitored without listening to the audio, and to easily provide an appropriate response depending on the situation. (Effects of Appendix 1) Parents can understand the content and intent without hearing the audio, respond quickly with a single tap, and those being monitored can instantly confirm the response.

[0110] According to this disclosure, voice messages from the person being monitored are transcribed into text by the server, and a summary and the speaker's intent (including urgency, emotion, request, etc.) are accurately generated and notified to the parent's device. Parents can instantly grasp the gist and intent of the message without listening to the audio. In addition, optimized response options are displayed on the push notification according to the speaker's intent and the parent's situation, allowing parents to respond quickly with simple operations such as a single tap.

[0111] This allows, for example, when a child is in a dangerous situation, parents to understand the situation within seconds and start a call or check the child's current location with a single tap. Compared to conventional systems, this significantly reduces the time it takes to respond and greatly contributes to ensuring the child's safety.

[0112] In addition, this disclosure can promote higher-quality communication and contribute to deepening the trust relationship between parent and child by conveying to parents the multifaceted intentions, such as emotions and requests, contained in the utterances of the person being cared for. Furthermore, by adjusting the notification method and amount of information according to the circumstances of the parents, it can reduce the cognitive and mental burden on parents and contribute to improving their work-life balance.

[0113] Furthermore, for those being monitored, the experience of having their intentions accurately conveyed and receiving appropriate feedback can help improve their communication skills, foster trust in their guardians, and encourage autonomous behavior. In the long term, the monitoring data accumulated through this system (appropriately anonymized and statistically processed) has the potential to contribute to the public interest, such as disseminating local safety information, gaining useful insights into children's behavior and psychology, and improving safety confirmation systems during disasters. Overall, this disclosure will have remarkable effects, including ensuring the safety and mental well-being of those being monitored, reducing the burden on guardians, and facilitating good communication between parents and children, thereby dramatically improving the quality and convenience of monitoring.

[0114] [Note 2] A transmission unit that sends voice data and current location information to a server, A receiving unit that receives response data from the server, including a response candidate code selected on the parent terminal, An output unit that performs at least one of the following: audio output, vibration output, and optical output, which are associated with the aforementioned candidate response code. A monitoring terminal equipped with the following features. (Issues corresponding to Appendix 2) The person being monitored must send information to the server and ensure that a response from their guardian is received. (Effects due to Appendix 2) Information can be sent, and responses from parents can be reliably received through multiple modal windows.

[0115] [Note 3] A voice processing unit that converts audio data received from a monitored device into text and generates a summary and the speaker's intent, A notification transmission unit that transmits notification data to the guardian's terminal, including a plurality of response candidate codes corresponding to the utterance intent, the summary text, and the location information of the monitored terminal. A response transmission unit that transmits a response candidate code received from the guardian terminal to the monitored terminal. A server equipped with the following features. (Issues corresponding to Appendix 3) Information from those being monitored needs to be processed and communicated in a way that is easy for guardians to understand and respond to. (Effects of Appendix 3) Audio data can be converted into information that is easy for parents to understand, encouraging quick responses.

[0116] [Note 4] The parent's computer, A display control unit that receives notification data from a server and displays a push notification containing a summary and multiple response candidate codes, and A reply unit that sends the selected response candidate code back to the server. A program designed to function as such. (Issues corresponding to Appendix 4) Parents need to receive notifications from the server and perform a simple response. (Effects of Appendix 4) Parents can receive notifications on their devices and respond with simple operations.

[0117] [Note 5] In the monitoring support system described in Appendix 1, The notification data includes an icon indicating the intended speech at the beginning of the summary text. (Issues corresponding to Appendix 5) I want to grasp the speaker's intent more intuitively. (Effects of Appendix 5) Icons allow you to grasp the urgency and type of speech at a glance.

[0118] [Note 6] In the monitoring support system described in Appendix 1, The parent device displays the location information as a map thumbnail image indicating the location, along with the summary text. (Issues corresponding to Appendix 6) I want to be able to instantly determine the location of the person being monitored. (Effects of Appendix 6) The message content and geographical context can be understood simultaneously.

[0119] [Note 7] In the monitoring support system described in Appendix 1, The parental device arranges the response buttons corresponding to the multiple response candidate codes in order of highest estimated selection rate based on the parental device's past selection history. (Issues corresponding to Appendix 7) I want to further streamline the process of selecting the response button. (Effects of Appendix 7) Buttons are optimized according to usage, reducing the user's workload.

[0120] [Note 8] In the monitoring support system described in Appendix 1, When the parental device receives a long tap operation on the push notification, it switches the push notification from the first display state to the second display state, and in the second display state, it displays a larger number of response buttons than in the first display state. (Issues corresponding to Appendix 8) Ideally, I'd like a simple display, but with plenty of response options available when needed. (Effects of Appendix 8) It can achieve both a simple UI and a wide range of options.

[0121] [Note 9] In the monitoring support system described in Appendix 1, If the unread notification data falls under a predetermined abnormality category, the parental device displays the notification icon associated with the push notification in a different color than usual. (Issues corresponding to Appendix 9) I want to make sure I don't miss any important notifications. (Effects of Appendix 9) This makes it easier to notice important notifications.

[0122] [Note 10] In the monitoring support system described in Appendix 1, The parent device displays a countdown bar indicating that a default response will be automatically sent if a predetermined time elapses without the response button being selected for the push notification. (Issues corresponding to Appendix 10) Even if I can't respond immediately, I want to be able to give feedback to the person being cared for. (Effects of Appendix 10) This can reduce the anxiety of the person being monitored during response delays.

[0123] [Note 11] In the monitoring support system described in Appendix 1, When the push notification accepts a selection operation, the parent terminal enables the playback of the corresponding voice message and transitions to an application screen that displays a map showing the location information. (Issues corresponding to Appendix 11) I want to be able to easily access detailed information (voice, map) from notifications. (Effects of Appendix 11) Detailed verification and access to operations will be improved.

[0124] [Note 12] In the monitoring support system described in Appendix 1, When the response button is selected, the parent terminal updates the notification data or the response button to a display format indicating that a response has been given. (Issues corresponding to Appendix 12) I want to clarify which notification I responded to. (Effects of Appendix 12) This makes it easier to manage response status and prevents duplicate responses.

[0125] [Note 13] In the configuration described in Appendix 1 or Appendix 3, the voice processing unit of the server further comprises an acoustic analysis unit that extracts acoustic features from the voice data and generates the speech intent and its urgency level based on the transcribed data and the acoustic features. (Issues corresponding to Appendix 13) I want to more accurately grasp the urgency of the utterance. (Effects of Appendix 13) By considering acoustic information such as tone of voice and non-verbal sounds in addition to text information, it becomes possible to more accurately determine the level of urgency, especially from unclear or emotional speech in children, and reduce the likelihood of missing important information.

[0126] [Note 14] In the configuration described in Appendix 1, Appendix 3, or Appendix 13, the notification generation unit of the server includes in the multiple response candidate codes action candidate codes (e.g., instructions to automatically make a call to the monitored terminal, instructions to display a map / route on the parent terminal, instructions to forward the notification to other relevant parties, etc.) to support the next specific action of the parent, depending on the intent of the utterance and / or the urgency level. (Issues corresponding to Appendix 14) We want to enable parents to take quicker and more appropriate action in emergencies. (Effects of Appendix 14) Parents can directly and quickly select and execute the most appropriate initial action based on the situation via push notifications, thus reducing response time.

[0127] [Note 15] In the configuration described in Appendix 1, Appendix 4, or Appendix 13, the display control unit of the parent terminal dynamically changes the display format of the push notification (e.g., icon type, notification background color, arrangement and number of response buttons, notification sound, vibration pattern) according to the utterance intent and / or urgency level included in the notification data. (Issues corresponding to Appendix 15) We want to make it possible for parents to see important information, such as the urgency of a situation, at a glance. (Effects of Appendix 15) Parents can intuitively grasp the importance and urgency of a message with just a glance at a push notification, or even when they are not looking at the screen, enabling them to make quick decisions and take action.

[0128] [Note 16] In the configuration described in Appendix 1, Appendix 4, or any of Appendix 13 to 15, the display control unit of the parent terminal displays, in the push notification, information indicating the main reason why the response button was presented, along with a response button corresponding to the plurality of response candidate codes. (Issues corresponding to Appendix 16) We want to clarify the reasons why response options were presented and increase transparency in the server's internal processing. (Effects of Appendix 16) This makes it easier for parents to judge the appropriateness of response options, and also allows external verification of the correspondence between the speaker's intent and the response options.

[0129] [Note 17] In the monitoring support system described in Appendix 1, at least a portion of the transcription of the voice data, generation of the summary text, generation of the utterance intent, or generation of the response candidate code is performed on the monitored terminal or the guardian terminal. (Issues corresponding to Appendix 17) We want to increase the flexibility of the system configuration and cover configurations other than server-centralized ones. (Effects of Appendix 17) The basic benefits of this disclosure can be enjoyed while ensuring the distribution of processing load and continued operation under specific communication environments.

[0130] [Note 18] In the configurations described in Appendix 1, Appendix 3, or Appendix 13, "generating speech intent" includes performing information processing based on speech data or information derived therefrom that reflects the speaker's situation, desires, or emotional state, and the result of that processing influences subsequent processing, and does not necessarily require a step of explicitly generating "speech intent" as a separate data item. (Issues corresponding to Appendix 18) We want to clarify the interpretation of the act of "generating speech intent" and avoid being limited to a specific architecture. (Effects of Appendix 18) Systems that functionally process intent internally, such as end-to-end models, can also be more easily included in the scope of rights.

[0131] [Note 19] In the configurations described in Appendix 1, Appendix 3, or Appendix 14, “Response Candidate Code” means identification information that indicates the type, content, or action of response selected by the parent, and includes any form of information that allows the system to identify the parent’s selection, such as numbers, strings, flags, or commands that instruct the transmission of specific data or the execution of an action. (Issues corresponding to Appendix 19) We want to cover a wider range of communication methods without limiting the format of the "candidate response code." (Effects of Appendix 19) This makes it easier to include forms such as free responses (transmission of actual data such as voice or text) within the scope of rights.

[0132] [Note 20] In the configurations described in Appendix 1 or Appendix 3, “summary” includes information that presents the main points of the transcribed audio data in a short format, or information that presents important parts of the original text in a format that makes them identifiable (e.g., keyword highlighting). (Issues corresponding to Appendix 20) We aim to expand the concept of "summary text" to cover diverse forms of information presentation that aid in comprehension. (Effects of Appendix 20) Systems that have essentially the same effect as summarization, such as full text display plus keyword highlighting, will also be more easily included in the scope of rights.

[0133] [Note 21] In the configurations described in Appendix 1 or Appendix 4, "push display" includes not only display via the OS standard push notification function, but also proactive information presentation in modal windows or dedicated areas within specific applications. (Issues corresponding to Appendix 21) We want to clarify the interpretation of "push notifications" and cover notification methods other than those standard OS functions. (Effects due to Appendix 21) This makes it easier to include systems that primarily rely on notification displays within dedicated applications within the scope of rights.

[0134] [Note 22] In the configuration described in any of Appendix 1, Appendix 3, Appendix 13, or Appendix 18, the voice processing unit of the server further generates, as the utterance intent, at least one of the emotion category information, action request information, or situation context information contained in the voice data. (Issues corresponding to Appendix 22) We want to capture the multifaceted aspects of a speaker's intent and enable more nuanced responses. (Effects due to Appendix 22) This enables the generation of more appropriate notification displays and response options, taking into account not only urgency but also emotions, requests, and the overall situation.

[0135] [Note 23] In the configuration described in any of Appendix 1, Appendix 3, Appendix 14, or Appendix 19, the notification generation unit of the server includes at least one of the following as the plurality of response candidate codes: intent-direct candidate, context-dependent candidate, learning-type candidate, action-execution-type candidate, and free-response-type candidate. (Issues corresponding to Appendix 23) We want to improve convenience for parents by diversifying the logic for generating response candidates. (Effects due to Appendix 23) It can present more personalized and useful response options tailored to the situation and the parents' preferences.

[0136] [Note 24] In the configuration described in Appendix 1 or Appendix 3, the server further comprises at least one of the following: history analysis and learning processing unit, parent status estimation unit, and notification optimization processing unit. (Issues corresponding to Appendix 24) We want to expand the server's functionality to provide more advanced monitoring support. (Effects due to Appendix 24) This will enable continuous system improvements, notifications tailored to the parents' situations, and more advanced warnings.

[0137] [Note 25] In the configuration described in Appendix 1 or Appendix 2, the output unit of the monitored terminal selects and executes at least one of the following based on the response candidate code: response transmission output, warning output, and emergency alarm output. (Issues corresponding to Appendix 25) We want to clarify the purpose of the output on the monitored terminal and improve the reliability of the response. (Effects due to Appendix 25) Responses and instructions from guardians are reliably communicated to the person being supervised, according to their purpose.

[0138] [Note 26] In the monitoring support system described in Appendix 1, the server further comprises a large-scale language model, the voice processing unit generates the summary text or supplementary explanatory text regarding the utterance intent using the large-scale language model, and the notification generation unit generates contextually appropriate response candidate messages based on the utterance intent and the guardian's past response history using the large-scale language model. (Issues corresponding to Appendix 26) We want to make notifications to parents more natural and easier to understand, and improve the diversity and appropriateness of response options. (Effects of Appendix 26) Parents will be able to understand the situation in a way that is closer to natural language, and will be able to easily provide more human-like, situation-appropriate, and nuanced responses.

[0139] [Note 27] In the monitoring support system described in Appendix 1, the history analysis and learning processing unit of the server learns and updates the intent determination model of the voice processing unit or the response candidate generation logic of the notification generation unit using a federated learning method without collecting local data stored on each terminal. (Issues corresponding to Appendix 27) We want to provide personalized monitoring support while highly protecting the privacy of those being monitored. (Effects of Appendix 27) This allows us to provide personalized monitoring features while offering stronger protection for user privacy.

[0140] [Note 28] In the monitoring support system described in Appendix 1, the monitored terminal further includes a biosensor that acquires the biometric information of the monitored person, the transmitting unit transmits the biometric information to the server, and the voice processing unit of the server uses the biometric information to determine the urgency level of the intended speech. (Issues corresponding to Appendix 28) We want to further improve the accuracy of our assessment of the urgency level of those being monitored and reduce the risk of overlooking a problem. (Effects due to Appendix 28) Even in cases where it is difficult to make a judgment based solely on the content of speech or tone of voice, objective biometric information can be used to more accurately assess the urgency and prevent serious oversights.

[0141] [Note 29] In the monitoring support system described in Appendix 1, the server records a monitoring log on the blockchain, which includes at least a portion of the voice data from the monitored terminal, the location information, the intended speech, and the candidate response codes from the guardian terminal. (Issues corresponding to Appendix 29) We want to ensure the reliability and non-tampering of important records related to monitoring. (Effects due to Appendix 29) The authenticity of logs such as emergency communications and location information can be guaranteed, and their evidentiary value can be enhanced when necessary.

[0142] [Note 30] An elderly monitoring system that analyzes the speech of elderly people and notifies caregivers of their intentions and health status, and comprises the configuration of the monitoring support system described in Appendix 1. (Issues corresponding to Appendix 30) In monitoring the elderly, it is necessary to accurately grasp the situation from audio information and support a swift response. (Effects of Appendix 30) Applying the technology disclosed herein to monitoring the elderly can contribute to ensuring the safety and health management of the elderly, as well as reducing the burden on caregivers. [Explanation of Symbols]

[0143] 1. Monitoring and support system 100... Monitoring device 101... Control Unit 102...Storage section 103... Communications Department 104...Mike 105...GPS receiver Output section of 106... 107...Operation unit 111...Voice acquisition unit 112...Location information acquisition unit 113...Transmitter 114... Receiver 115...Output Control Unit 200... Server 201... Control Unit 202...Storage section 203... Communications Department 211... Receiver 212...Audio Processing Unit 212a...Speech-to-text conversion section 212b...Summary sentence generation unit 212c... Speech intent generation unit 212d...Text Analysis Department 212e...Acoustic analysis department 213...Notification generator 214...Transmitter 215...Response Processing Unit 216...History Analysis / Learning Processing Unit 217...Parental Status Estimation Department 218...Notification Optimization Processing Unit 300...Parental device 301... Control Unit 302...Storage section 303... Communications Department 304...Display section 305...Operation unit 306...Speaker 311... Receiver 312...Display Control Unit 313...Operation Reception Section 314...Sending section (Reply section) 500A, 500B... Push notifications 501... Icon 502...Name 503...Summary 504...Map thumbnail image 505A~505D...Response button 506... Countdown bar 900...Notification data 901...Message ID 902... Timestamp 903... Person being monitored ID 904...Location information 905...Audio data (or reference thereof) 906...Text data 907...Summary 908... Speech intent code 909...List of candidate response codes 910... Abnormal Classification Flag NW... Communication Network

Claims

1. Includes the monitored device, server, and guardian device, The monitored terminal comprises a transmitting unit that transmits acquired voice data and current location information to the server, a receiving unit that receives response data from the server including a candidate response code selected by the guardian terminal, and an output unit that performs a predetermined output based on the response data. The server comprises: an audio processing unit that converts the audio data received from the monitored terminal into text to generate a summary and a speech intent; a notification transmission unit that transmits notification data including a plurality of response candidate codes corresponding to the speech intent, the summary, and the location information of the monitored terminal to the guardian terminal; and a response transmission unit that transmits the response candidate codes received from the guardian terminal to the monitored terminal. The parent terminal includes a display control unit that receives the notification data from the server and displays a push notification including the summary text and the multiple response candidate codes, and a reply unit that sends back the selected response candidate code to the server. A monitoring and support system.

2. A transmission unit that sends voice data and current location information to a server, A receiving unit that receives response data from the server, including a response candidate code selected on the parent terminal, An output unit that performs at least one of the following: audio output, vibration output, and optical output, which are associated with the aforementioned candidate response code. A monitoring terminal equipped with the following features.

3. A voice processing unit that converts audio data received from a monitored device into text and generates a summary and the speaker's intent, A notification transmission unit that transmits notification data to the guardian's terminal, including a plurality of response candidate codes corresponding to the utterance intent, the summary text, and the location information of the monitored terminal. A response transmission unit that transmits a response candidate code received from the guardian terminal to the monitored terminal. A server equipped with the following features.

4. The parent's computer, A display control unit that receives notification data from a server and displays a push notification containing a summary and multiple response candidate codes, and A reply unit that sends the selected response candidate code back to the server. A program designed to function as such.

5. In the monitoring support system described in claim 1, The notification data includes an icon indicating the utterance intent at the beginning of the summary sentence. A monitoring and support system.

6. In the monitoring support system described in claim 1, The parental device displays the location information as a map thumbnail image indicating the location, along with the summary text. A monitoring and support system.

7. In the monitoring support system described in claim 1, The parental device arranges the response buttons corresponding to the multiple response candidate codes in order of highest estimated selection rate based on the parental device's past selection history. A monitoring and support system.

8. In the monitoring support system described in claim 1, When the parental device receives a long tap operation on the push notification, it switches the push notification from the first display state to the second display state, and in the second display state, it displays a larger number of response buttons than in the first display state. A monitoring and support system.

9. In the monitoring support system described in claim 1, The parental device, if the unread notification data falls under a predetermined abnormality category, displays the notification icon associated with the push notification in a different color than usual. A monitoring and support system.

10. In the monitoring support system described in claim 1, The parent device displays a countdown bar indicating that a default response will be automatically sent if a predetermined time elapses without the response button being selected for the push notification. A monitoring and support system.

11. In the monitoring support system described in claim 1, When the push notification receives a selection operation, the parent terminal enables the playback of the corresponding voice message and transitions to an application screen that displays a map showing the location information. A monitoring and support system.

12. In the monitoring support system described in claim 1, When the response button is selected, the parent terminal updates the notification data or the response button to a display format indicating that the response has been given. A monitoring and support system.

Images