A monitoring system, monitoring method, and program equipped with voice-based danger detection and notification functions.

Abstract

This technology provides a way to quickly notify caregivers of danger when a person being monitored, such as a child, encounters a sudden accident or incident and is unable to call for help themselves. [Solution] A monitoring device carried by the person being monitored uses a microphone to acquire ambient acoustic data. The monitoring device or server detects the person being monitored in a dangerous state based on the acoustic data. When a dangerous state is detected, the server sends an emergency notification to the monitoring device carried by the person being monitored, in a manner different from normal notifications, which includes disabling some of the user settings of the monitoring device. Dangerous state detection is performed using volume, specific keywords, frequency characteristics, or machine learning models, and the emergency notification may include location information of the monitoring device or automatically establish a two-way voice communication channel.

Description

Technical Field

[0001] The present invention relates to a monitoring system, a monitoring method, and a program for confirming the current position and safety of a person under guardianship. In particular, the present invention relates to a technology for automatically detecting the danger of a person under guardianship based on the voice acquired by a terminal for the person under guardianship and notifying the guardian.

Background Art

[0002] In recent years, a so-called monitoring service has become widely popular, in which a mobile terminal equipped with a positioning function such as GPS (Global Positioning System) and a communication function is possessed by a child or an elderly person who may wander (hereinafter collectively referred to as "person under guardianship" in this specification), and a guardian or family member (hereinafter collectively referred to as "guardian" in this specification) can confirm the current location of the person under guardianship in real time on their own smartphone or the like.

[0003] This type of service is generally configured as a client-server system including a terminal for the person under guardianship possessed by the person under guardianship, a monitoring terminal possessed by the guardian, and a server for the monitoring service that mediates communication between the two and manages various data. The terminal for the person under guardianship is equipped with GNSS (Global Navigation Satellite System) positioning means, acquires its own position information at predetermined intervals, and transmits it to the server via a communication network such as a cellular network. As a result, the guardian can always confirm the current location and past movement history of the person under guardianship on the map of a dedicated application (monitoring application) installed on their own monitoring terminal.

[0004] Furthermore, many monitoring devices are equipped with microphones and speakers, enabling the sending and receiving of voice messages. For example, the monitored device has input sections such as physical buttons and touch interfaces that are simplified as much as possible so that even children can easily operate them. By operating these input sections, the monitored person can send recorded voice messages to their caregiver. This allows the monitored person to contact their caregiver or proactively ask for help if they encounter any trouble, such as getting lost or feeling unwell.

[0005] Related literature on these background technologies includes, for example, Patent Documents 1-8 and Non-Patent Document 1. These technologies provide a certain level of reassurance to the caregiver and contribute to ensuring the safety of the person being cared for. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] WO2023 / 162120 [Patent Document 2] Patent No. 7505832 [Patent Document 3] Patent No. 7508156 [Patent Document 4] Patent No. 7525968 [Patent Document 5] Patent No. 7525971 [Patent Document 6] Patent No. 7664664 [Patent Document 7] Patent No. 7680105 [Patent Document 8] Patent No. 7696671 [Non-patent literature]

[0007] [Non-Patent Document 1] B-Size Co., Ltd., BoT Talk [Overview of the project] [Problems that the invention aims to solve]

[0008] According to the conventional technology described above, the caregiver can gain a certain level of reassurance by constantly monitoring the location of the person being cared for. Furthermore, if any trouble occurs with the person being cared for, they can use a voice message function or similar to ask the caregiver for help.

[0009] However, conventional technologies assumed that the person being monitored would operate the device of their own volition. Therefore, in sudden and imminent dangerous situations, such as being involved in a sudden accident or being attacked by a suspicious person who might abduct or kidnap someone, it is conceivable that the person being monitored, such as a child, might panic, resist, or lose consciousness due to the shock, making it extremely difficult for them to take proactive action to call for help.

[0010] Conventional technologies have not adequately provided a mechanism for the system to autonomously detect danger when the person being monitored becomes passive and to quickly warn the monitor. In other words, there was a problem in that monitors could not immediately know of the occurrence of danger unless they received contact from the person being monitored or inferred an anomaly from indirect information such as the location information on the map not changing for a long period of time.

[0011] This invention has been made in view of the above circumstances, and aims to provide a new monitoring system, monitoring method, and program that utilize the microphone sound collection of the monitored terminal to quickly and autonomously detect imminent danger to the person being monitored, reliably and effectively notify the caregiver, and enable immediate action. [Means for solving the problem]

[0012] To achieve the above objective, a monitoring method according to one aspect of the present invention is a method performed by a monitoring system including a portable monitored terminal equipped with a battery, a monitoring terminal, and a server, characterized in that the monitored terminal acquires ambient acoustic data using its microphone; the monitored terminal or the server detects a predetermined dangerous state based on the acoustic data, indicating the possibility that the monitored person is in danger; and in response to the detection of the dangerous state, the server sends an emergency notification to the monitoring terminal, in a manner different from a normal notification, which includes a process to disable a part of the user settings on the monitoring terminal.

[0013] Furthermore, a monitoring system according to one aspect of the present invention is a monitoring system comprising a portable monitored terminal equipped with a battery, a monitoring terminal, and a server, wherein the monitored terminal is equipped with a microphone, the monitored terminal or the server detects a dangerous situation based on acoustic data acquired by the microphone, and in response to the detection of the dangerous situation, the server is configured to send an emergency notification to the monitoring terminal, which includes processing to disable a part of the user settings on the monitoring terminal.

[0014] Furthermore, a program according to one aspect of the present invention is for causing a computer to function as the server or the monitored terminal in the above-mentioned monitoring system.

[0015] With this configuration, even when the person being monitored is unable to perform any active operations, the system can autonomously detect the occurrence of danger based on objective information such as sounds around the monitored device. When danger is detected, the system forcibly overrides the notification settings of the monitoring device (e.g., silent mode or do not disturb mode), ensuring that the person being monitored does not miss the notification and that a warning is delivered reliably and immediately. [Effects of the Invention]

[0016] According to the present invention, by utilizing the microphone sound collection of the monitored terminal, it is possible to quickly detect an imminent dangerous situation in which the monitored person cannot operate by themselves and reliably and effectively notify the monitor. As a result, the monitor can be aware of the occurrence of danger at an early stage, immediately grasp the situation, and take prompt and appropriate initial actions such as rushing to the scene, reporting to the police or emergency services, which can significantly enhance the safety of the monitored person.

Brief Description of the Drawings

[0017] [Figure 1] It is a block diagram showing the overall configuration of a monitoring system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing an example of the hardware configuration of the monitored terminal. [Figure 3] It is a block diagram showing an example of the hardware configuration of the monitoring terminal. [Figure 4] It is a block diagram showing an example of the hardware configuration of the server. [Figure 5] It is a functional block diagram showing the functional configurations of the monitored terminal and the server. [Figure 6] It is a flowchart showing the sequence of normal voice message transmission and reception processing (first notification) in this embodiment. [Figure 7] It is a flowchart showing the sequence of emergency notification processing (second notification) in this embodiment. [Figure 8] It is a flowchart showing an example of a subroutine of danger state detection processing. [Figure 9] It is a conceptual diagram of danger state detection processing by a machine learning model. [Figure 10] It is a flowchart showing the sequence of automatic voice call path establishment processing in an emergency notification. [Figure 11] It is a diagram showing an example of a talk screen on the monitoring terminal. [Figure 12] It is a diagram showing an example of the display of the first notification on the monitoring terminal. [Figure 13]This figure shows an example of the display of the second notification on a monitoring device (part 1). [Figure 14] This figure shows an example of the display of the second notification on a monitoring device (part 2), including location information. [Figure 15] This is a conceptual diagram illustrating continuous monitoring and control based on the distance between the monitoring terminal and the monitored terminal. [Figure 16] This is a conceptual diagram illustrating continuous monitoring and control based on a set schedule. [Figure 17] This figure shows an example of user mapping information managed on a server. [Figure 18] This flowchart shows a modified example of the hazardous state detection process in a server. [Modes for carrying out the invention]

[0018] Hereinafter, embodiments of the present invention (hereinafter, "these embodiments") will be described in detail with appropriate reference to the drawings. In each figure, the same or corresponding parts are denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, unless otherwise specified, the components described herein can be implemented individually or in combination with other components.

[0019] <1. System Configuration> Figure 1 is a block diagram showing the overall configuration of a monitoring system 1 according to one embodiment of the present invention. The monitoring system 1 comprises one or more monitoring terminals 10 each possessed by one or more persons being monitored B, one or more monitoring terminals 20 each possessed by one or more monitors A, and a server 30 that is connected to each of these terminals via a communication network NW.

[0020] The communication network NW is a wide-area communication network, either wired or wireless, including cellular networks provided by mobile phone carriers such as LTE (Long Term Evolution) and 5G (5th Generation Mobile Communication System), Wi-Fi (Wireless Fidelity) networks, and the internet. The monitored terminal 10 and the monitoring terminal 20 communicate data with the server 30 via this communication network NW. The communication protocol is based on TCP / IP, and at the application layer, HTTP / HTTPS, WebSocket, MQTT (Message Queuing Telemetry Transport), etc., are used as appropriate depending on the application. For example, HTTPS may be used for periodic information transmission from the terminal, while WebSocket or MQTT may be used for real-time push notifications and bidirectional communication from the server to the terminal.

[0021] The monitored device 10 is a battery-powered portable electronic device intended for daily use by the person being monitored, B, such as a school-aged child or an elderly person at risk of wandering. The monitored device 10 should be small and lightweight so that the person being monitored, B, can carry it without discomfort or burden. For example, its casing should be small enough to fit in a child's palm (e.g., approximately 5cm x 5cm x 2cm thick) and preferably have waterproof and dustproof properties (such as IP67). The casing material should be lightweight and highly impact-resistant, such as ABS resin or polycarbonate. It may also have an attachment mechanism that allows it to be hung from the neck with a strap, or attached to a bag, backpack, belt, or clothing using a clip, hook, or strong magnet.

[0022] The monitoring device 20 is an electronic device owned by caregiver A, and is typically a smartphone running an operating system such as iOS® or Android®. However, it is not limited to these, and may also be a tablet device, smartwatch, personal computer (PC), etc. The monitoring device 20 has dedicated application software (hereinafter referred to as the monitoring app) for using this monitoring service downloaded and installed from an application store such as the App Store® or Google Play®. Caregiver A uses this monitoring app to use the various functions and change settings described later.

[0023] Server 30 consists of one or more physical or virtual computers that centrally manage the entire monitoring system 1. Server 30 may be built on a cloud computing environment (e.g., Amazon Web Services®, Google Cloud Platform®, Microsoft Azure®, etc.). This ensures scalability, availability, and maintainability. Server 30 performs various backend processes necessary for providing the monitoring service, including web server and API server functions that relay the transmission and reception of various information between the monitored terminal 10 and the monitoring terminal 20, as well as user authentication, user information management, terminal mapping management, location information storage and management, primary storage of voice data, and billing processing. Data is transmitted and received in formats such as JSON (JavaScript Object Notation) and Protocol Buffers.

[0024] Figure 17 shows an example of user mapping information 361 stored in the storage unit 32 of the server 30. The user mapping information 361 is configured as a table in a relational database or NoSQL database. In this example, it has columns such as "monitoring terminal ID", "owner", "monitored terminal ID", and "owner". For example, it is possible to set it so that the monitoring terminal with ID "001" (owner: mother) has the authority to monitor both the monitored terminal with ID "A1" (owner: Taro) and the monitored terminal with ID "A2" (owner: Jiro). Similarly, it is possible to flexibly set up many-to-many relationships such as the monitored terminal with ID "A1" being monitored by both the monitoring terminals with ID "001" (mother) and ID "002" (father). This information is referenced when determining the destination of various notifications, which will be described later.

[0025] <2. Hardware Configuration> Figure 2 is a block diagram showing an example of the hardware configuration of the monitored terminal 10. The monitored terminal 10 is configured as a computer system in which a processor, memory, communication module, various sensors, etc. are interconnected via a bus 19. Specifically, it includes a control unit 11, a storage unit 12, a communication unit 13, a GNSS receiver 14, an input unit 15, a microphone 16, a speaker 17, a vibrator 18, etc. These are mounted on a printed circuit board, housed in a plastic or metal casing, and powered by a built-in rechargeable lithium-ion battery or other battery (not shown).

[0026] The control unit 11 consists of a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or an SoC (System-on-a-Chip), and comprehensively controls the operation of the entire monitored terminal 10 by executing the operating system and various programs stored in the memory unit 12. In particular, a processor with an ARM® architecture, which prioritizes power saving, is preferably used. The memory unit 12 includes non-volatile ROM (Read Only Memory) and flash memory, and volatile RAM (Random Access Memory). The ROM and flash memory store firmware and programs executed by the control unit 11, individual identification IDs, and various setting data. The RAM is used as the working memory of the control unit 11 and holds temporary data during processing. The communication unit 13 is a communication interface for connecting to a communication network NW, and includes a communication module compatible with cellular communication standards such as LTE or 5G, and an antenna. It is also possible to use modules compatible with low-power communication standards such as LTE-M (Cat-M1) or NB-IoT. The GNSS receiver 14 receives positioning signals from GNSS satellites such as GPS, GLONASS, Galileo, and Quasi-Zenith Satellite System (QZSS), and calculates position information including the aircraft's latitude, longitude, and altitude. To improve positioning accuracy, it may support A-GPS (Assisted GPS) and acquire satellite orbit information from the server 30. In addition, in places where GNSS signals cannot reach, such as indoors, auxiliary positioning functions using Wi-Fi access point signal strength or BLE (Bluetooth Low Energy) beacons may be used in combination. The input unit 15 is an interface for receiving operation input from the person being monitored B, and can be, for example, a physical push button, a capacitive touch sensor, or a small touch panel display. To prevent accidental operation, measures may be taken to detect a long press for a certain period of time and activate a specific function. Microphone 16 is a MEMS (Micro Electro Mechanical Systems) microphone, which collects ambient sound and converts it into an analog electrical signal. This analog signal is converted into digital audio data by an A / D converter in an audio codec IC (not shown) and processed by the control unit 11. Speaker 17 converts the analog signal output from the D / A converter within the audio codec IC into audio and outputs it. The vibrator 18 is an eccentric motor or a linear oscillator, and its vibration provides tactile notification to the person being monitored B. In addition to these, acceleration sensors, gyroscopes, geomagnetic sensors, and temperature sensors may be installed, and the information from these sensors may be used as an aid in determining dangerous conditions. For example, if the timing of the acceleration sensor detecting a large impact (such as a fall) coincides with the timing of the detection of a loud sound, it can be determined that there is a high probability of danger.

[0027] Figure 3 is a block diagram showing an example of the hardware configuration of the monitoring terminal 20. If the monitoring terminal 20 is a smartphone, its configuration is publicly known, so a detailed explanation will be omitted, but it includes a control unit 21 which is a processor, a storage unit 22 which is memory, a communication unit 23 which performs cellular communication and Wi-Fi communication, a GNSS receiver unit 24 which acquires location information, a display unit 25 which displays a GUI, an input unit 26 such as a touch panel, a microphone 27 for voice input, and a speaker 28 for voice output.

[0028] Figure 4 is a block diagram showing an example of the hardware configuration of server 30. Server 30 has the configuration of a typical server computer, comprising a control unit 31 which is one or more processors, a storage unit 32 which includes main memory and storage (HDD, SSD), a communication unit 33 which includes a network interface card (NIC), and an input / output interface 34 for console connection.

[0029] <3. Functional Configuration and Processing> Next, the functional configuration of the monitoring system 1 according to this embodiment and its processing details will be explained in detail with reference to the functional block diagram in Figure 5 and other figures.

[0030] The monitored terminal 10 has a functional configuration that includes a terminal control unit 110, an acoustic data acquisition unit 120, a location information acquisition unit 130, and a notification processing unit 140. These are mainly realized by the control unit 11 of the monitored terminal 10 executing a program stored in the storage unit 12. The server 30 has a functional configuration consisting of a server control unit 310, a communication relay unit 320, a user management unit 330, and a risk determination unit 340. These are mainly realized by the control unit 31 of the server 30 executing a program stored in the storage unit 32.

[0031] <3-1. Basic Functions> First, we will explain the basic functions of the monitoring system 1, which are a prerequisite for understanding the present invention.

[0032] (location information notification function) The location information acquisition unit 130 of the monitored terminal 10 controls the GNSS receiving unit 14 based on instructions from the terminal control unit 110 to intermittently acquire the terminal's location information at predetermined intervals (for example, every minute, which can be changed by setting). The acquired location information, along with a timestamp, is transmitted to the server 30 via the communication unit 13 and the communication network NW. The communication relay unit 320 of the server 30 associates the received location information with the identifier (ID) of the source monitored terminal 10 and stores it chronologically in the location information database in the storage unit 32.

[0033] When caregiver A operates a monitoring application running on monitoring terminal 20 and requests the display of the current location of a specific person being monitored B, monitoring terminal 20 sends the request to server 30. After authentication and authorization by the user management unit 330, the server control unit 310 of server 30 searches the location information database for the latest location information corresponding to the requested monitored terminal ID and sends it to monitoring terminal 20. Based on the received location information, monitoring terminal 20 uses a map plotting library (e.g., Google Maps API) to plot and display an icon indicating the current location of person being monitored B on a map. This allows caregiver A to accurately know where person being monitored B is at any given time.

[0034] (Voice messaging function: First notification) Figure 6 shows the processing sequence when sending and receiving a normal voice message. When the person being monitored, B, operates the input unit 15 of the monitored terminal 10 (for example, by long-pressing the talk button), the terminal control unit 110 detects this and instructs the acoustic data acquisition unit 120 to record the audio. The acoustic data acquisition unit 120 records the audio input from the microphone 16 as digital audio data (for example, PCM data) at a predetermined sampling frequency (e.g., 16kHz) and quantization bit depth (e.g., 16bit), and stores it in the temporary area of ​​the storage unit 12 until a predetermined upper limit time (e.g., 30 seconds) is reached. When the button is released or the upper limit time is reached, the recording ends, and the recorded audio data is encoded into an irreversible compression format such as AAC (Advanced Audio Coding) or Opus to generate a single audio data file.

[0035] Once recording is complete, the terminal control unit 110 transmits the generated recording data to the server 30 via the communication unit 13 (step S100). At this time, identification information such as its own terminal ID is also transmitted. When the communication relay unit 320 of the server 30 receives the recorded data, it refers to the user mapping information 361 based on the terminal ID of the sender to identify one or more monitoring terminals 20 to which the notification should be sent, and forwards the recorded data to each monitoring terminal 20 (step S101). The transfer in step S101 may be performed on all monitoring terminals 20 of one or more caregivers A associated with the person being monitored B. Alternatively, the transfer in step S101 may be performed only on the monitoring terminals 20 of some of the caregivers A who are pre-configured from among the one or more caregivers A associated with the person being monitored B. For example, the transfer may be set to be performed only on the monitoring terminal 20 of caregiver A who is the mother. Alternatively, the transfer in step S101 may be performed only on the monitoring terminals 20 of some of the caregivers A who are dynamically determined from among the one or more caregivers A associated with the person being monitored B. For example, the transfer may be set to be performed only on monitoring terminals 20 that are in a predetermined positional relationship with the person being monitored 10 (for example, closer than a predetermined distance from each other, or further than a predetermined distance from each other) at the time the voice data is sent to the server 30 in step S100. Alternatively, for example, the transfer may be set to be performed on different monitoring terminals 20 depending on the time period in which the voice data is sent to the server 30 in step S100. For example, if it's in the morning, the data is transferred to the mother's monitoring device 10, and if it's in the afternoon, it's transferred to the father's monitoring device 10.

[0036] The monitoring terminal 20, upon receiving the recorded data, stores it in its internal storage unit 22 in a playable state and also sends a "first notification" to the caregiver A informing them that a voice message has arrived from the person being cared for B (step S102). The first notification is often implemented using the push notification function provided by the operating system. As shown in Figure 12, the sender of the message (Taro), the time of receipt, text such as "Message available," and an icon prompting playback are displayed at the top of the smartphone screen. This first notification may be displayed without sound or vibration, as it conforms to the current notification settings of the monitoring terminal 20 (such as silent mode, vibrate mode, or focus mode).

[0037] When caregiver A selects this first notification by tapping the banner (step S103), the monitoring app launches in the foreground and the corresponding recorded data is played from speaker 28 (step S104). Furthermore, a chat screen like the one shown in Figure 11 is displayed. The chat screen has a UI similar to that of a typical messaging app, and the history of voice messages sent and received so far is displayed chronologically along with the sender and time. You can also play any voice by tapping the icon of each message. It is also possible to send a voice message from the monitoring device 20 to the monitored device 10 using the same operation.

[0038] <3-2. Core Function of the Invention (Emergency Mode)> Next, we will explain in detail the core components of this invention: the detection of a dangerous situation and the accompanying emergency notification (second notification).

[0039] Figure 7 shows the processing sequence when emergency mode is activated and emergency notification processing is performed. In this embodiment, the acoustic data acquisition unit 120 of the monitored terminal 10 acquires ambient acoustic data based on instructions from the terminal control unit 110. This acquisition of acoustic data is not limited to the recording of the voice message described above, but also includes a "continuous monitoring mode" that is performed continuously or intermittently at predetermined intervals, regardless of input operations from the monitored person B, depending on the settings of the smart operation function described later.

[0040] (Detection of dangerous conditions) The danger determination unit 340 of the server 30, or the danger determination function of the monitored terminal 10 itself, determines whether the acquired acoustic data meets predetermined "specified conditions" (step S200). If these "specified conditions" are met, it is determined that there is danger to the monitored person B. This determination process is preferably performed by the server 30 from the standpoint of computing resources, but it can also be performed on the monitored terminal 10 side to avoid network delays (as described later in the modified example). Below, we will mainly describe an example in which the server 30 is the main determination entity.

[0041] Figure 8 shows an example of a subroutine for this hazardous condition detection process (step S200). The risk assessment body combines multiple criteria to make a comprehensive judgment about the risk situation. First, the volume level of the acquired acoustic data, specifically the RMS value and peak level, is analyzed, and it is determined whether or not the value exceeds a predetermined threshold (for example, 85 dB, which corresponds to the sound of a human scream or objects colliding violently) (step S211).

[0042] Next, speech recognition (ASR: Automatic Speech Recognition) is performed on the audio data and converted into text data using a speech recognition engine. Then, it is determined whether or not the text data contains keywords that have been pre-registered as indicating danger (for example, "help," "no," "stop," "it hurts," "thief," etc.) (step S212).

[0043] Furthermore, the frequency spectrum of the acoustic data is analyzed using a Fast Fourier Transform (FFT) or the like to determine whether its frequency characteristics match or are similar to the characteristics of acoustic patterns that have been pre-registered as indicating dangerous events (step S213). These acoustic patterns include patterns of high-frequency components unique to human screams, combination patterns of impact sounds and high-frequency shattering sounds when glass breaks, and patterns of sounds such as the sound of a car braking suddenly or colliding.

[0044] The risk assessment entity determines that a predetermined condition has been met and a risky situation has occurred if at least one of these assessment steps S211 to S213 is determined to be YES (step S214). These conditions may be used individually or in combination. For example, by logically combining multiple conditions, such as determining a risk only when "the volume is above a threshold" AND ("the keyword is included" or "the frequency characteristics match"), false positives, such as when a child is playfully shouting, can be effectively reduced, and the accuracy of detection (precision and recall) can be improved.

[0045] (Detection using machine learning models) In a more advanced embodiment, the detection of a hazardous state may be performed using a machine learning model trained specifically for a particular acoustic event. As shown in Figure 9, the hazard determination unit 340 includes a machine learning model 341. This machine learning model 341 is composed of, for example, a deep neural network (DNN), a recurrent neural network (RNN) that is robust to time-series data such as speech, or a convolutional neural network (CNN) that treats acoustic spectrograms as images, or a combination thereof.

[0046] Model 341 is pre-trained using a corpus consisting of a large amount of audio data. This training corpus includes audio data from various situations specific to children, collected and labeled specifically for the purposes of the present invention. Specifically, in addition to "non-dangerous" data such as normal speech, laughter, and play sounds, it includes a large amount of "dangerous" data that suggests a dangerous situation, such as genuine cries for help, screams of fear, groans in a panic, and utterances containing specific keywords ("help," "scary," etc.).

[0047] The danger determination unit 340 extracts acoustic features such as Mel-frequency cepstrum coefficients (MFCCs), spectrograms, zero crossing rates, and spectral centroids from the input raw acoustic data, and inputs these as feature vectors into a pre-trained machine learning model 341. Based on the input feature vectors, the model 341 outputs a probability between 0 and 1 that the sound is in a "dangerous state". If this probability exceeds a predetermined confidence threshold (e.g., 0.9), it is determined that the sound is in a dangerous state. By using such a machine learning model, it becomes possible to identify complex and subtle sound patterns in various situations that would be difficult to capture with simple rule-based determination, enabling extremely high-precision danger detection.

[0048] (Emergency notification: 2nd notification) Returning to Figure 7, if a dangerous situation is detected in step S200 (YES), the process switches to emergency mode. The monitored terminal 10 sends the recorded audio data (voice indicating danger detection) to the server 30, along with a flag indicating that this is an emergency mode, which is different from normal (for example, by setting the message priority header to "high") (step S201). On the other hand, if no dangerous situation is detected (NO), the normal voice message processing shown in Figure 6 (from step S101 onwards) is performed. In step S201, the monitored terminal 10 may transmit information indicating its current location at that time (the latest location information determined by the GNSS receiver 14) to the server 30 along with the voice data.

[0049] When server 30 receives data flagged with the emergency mode flag, it recognizes it as an emergency requiring top priority processing and sends a high-priority "second notification" to the monitoring terminal 20, which is clearly distinguishable from the normal first notification (step S202). This notification is delivered using high-priority notification mechanisms provided by the OS, such as Apple Push Notification service (APNs) "Critical Alerts" or Android's "Do Not Disturb override". The second notification in step S202 may include the current location information sent by the monitored terminal 10 in step S201.

[0050] Upon receiving the second notification, the monitoring terminal 20 immediately performs a powerful notification process to alert caregiver A of the danger (step S203). Compared to a normal first notification, this second notification provides a multifaceted and multi-layered notification experience having at least one of the following features.

[0051] 1. Forced Sound and Vibration Activation: The second notification forcibly disables (overrides) user settings such as silent mode or sleep mode set on the monitoring terminal 20, and activates a predetermined loud warning sound or strong vibration. This maximizes the likelihood that the monitor will notice the notification even when they are in a meeting or sleeping at night.

[0052] 2. Visually emphasized display: As shown in Figures 13 and 14, the display of the second notification is more visually emphasized than the normal first notification (Figure 12). For example, warning colors (such as red or yellow) are used across the entire screen, text such as "Emergency Message" or "SOS" is displayed in a larger font size than usual, and warning icons such as the exclamation mark "!" are made to flash, making it immediately clear that it is an emergency.

[0053] 3. Simultaneous display of location information: The second notification automatically includes the current location information of the monitored terminal 10 at the time the danger is detected. As shown in Figure 14, the monitoring terminal 20 simultaneously displays a map showing the current location of the monitored person B (Taro) in a modal window or in the notification's detailed view, along with a display indicating the emergency. This allows the monitor A to immediately grasp the nature and location of the danger and take concrete next actions such as "rushing to the scene" or "reporting the exact location to the police." Here, the current location information that is attached (displayed) should be based on the current location information included in the second notification (the current location information transmitted by the monitored terminal 10 along with the voice data in step S201).

[0054] 4. Persistent Notification: The warning sound and vibration triggered by the second notification do not end after a single instance, but continue repeatedly at regular intervals (for example, every 10 seconds) until caregiver A performs some explicit confirmation action, such as tapping the notification screen or operating a button on the screen. This ensures that caregivers will be aware of the notification later, even if they temporarily leave the device.

[0055] 5. Playback of detection sound: Instead of a generic warning sound provided by the system, the system plays the acoustic data used to detect the dangerous situation, i.e., the actual voice or scream uttered by the person being monitored, such as "Help!", as the notification sound for the second notification. Instead of displaying a text notification and then playing the acoustic data based on the selection of this notification, the system plays the acoustic data directly as the notification sound for the second notification, allowing the emergency to be immediately communicated to the caregiver A.

[0056] 6. Simultaneous transmission to all monitoring terminals: If multiple caregivers A (e.g., mother, father, grandparents) are registered for the person being monitored B, even if individual settings are possible for the normal first notification, such as "notify only the monitoring terminal of the mother, who is the primary contact," the second notification ignores all of these settings and is forcibly sent simultaneously to all monitoring terminals 20 registered in the user mapping information 361. In other words, as described in paragraph 0035 above, even if the first notification is set to be sent only to some of the monitoring terminals 20, this setting is disabled and the second notification is sent to each of the monitoring terminals 20 of each caregiver A associated with the person being monitored B. This maximizes the number of people who can respond, increases the likelihood that someone else can respond even if one person is unable to, and improves the certainty of rescue.

[0057] When caregiver A responds to this strong second notification by performing a notification playback operation (step S204), the recorded audio of the moment when danger was detected is played. As mentioned above, when playing audio data as the notification sound for the second notification, this step S204 is not necessary.

[0058] (Establishment of automated voice call channels) Figure 10 shows an example of additional processing during the second notification. In this embodiment, in order to further enhance the effectiveness of the emergency notification, processing to automatically establish a two-way voice call path may be performed after the transmission of the second notification. After sending a second notification to the monitoring terminal 20 (after step S202), the server 30 immediately begins the process of establishing a communication session between the monitoring terminal 20 and the monitored terminal 10 where danger has been detected (step S205). This process is carried out using VoIP (Voice over IP) technology such as SIP (Session Initiation Protocol) or WebRTC (Web Real-Time Communication). The server 30 functions as a signaling server and mediates the peer-to-peer (P2P) connection between the two terminals.

[0059] Once a session is established, two-way voice communication becomes possible between the two terminals (step S206). This entire process should ideally be performed automatically without any additional actions by caregiver A, such as "making a phone call." This allows caregiver A to immediately hear the sounds around the monitored terminal 10 as soon as they receive the second notification (the so-called listening function). If necessary, caregiver A can also speak directly to the monitored B via the microphone of the monitoring terminal 20, giving them instructions or making appeals. This function dramatically improves the speed and accuracy of situation assessment, enabling more appropriate responses.

[0060] <3-3. Smart Operation Function> In light of the fact that the monitored terminal 10 is a battery-powered portable terminal, the monitoring system 1 according to this embodiment is equipped with a "smart operation function" to streamline its operation and reduce concerns about battery consumption and privacy. This function is particularly effective in the "continuous monitoring mode," which continuously or intermittently acquires ambient acoustic data to monitor for hazards.

[0061] (Distance-based monitoring and control) As shown in Figure 15, the terminal control unit 110 of the monitored terminal 10 can dynamically control the ON / OFF status of the continuous monitoring function based on the physical distance to the monitoring terminals 20. Specifically, the monitored terminal 10 periodically acquires location information of one or more associated monitoring terminals 20 via the server 30. It then compares its own location information with the location information of each monitoring terminal 20 and calculates the distance D to the nearest monitoring terminal 20. If this distance D is closer than a predetermined threshold Dth (e.g., 10m) (D≦Dth), it determines that there is a very high probability that the caregiver A and the monitored person B are together, and automatically switches the continuous monitoring function OFF (or to low-frequency mode). On the other hand, if the distance D to all monitoring terminals 20 is greater than the threshold (D>Dth), the continuous monitoring function is automatically turned ON. This stops unnecessary monitoring while the user is within the caregiver's sight, intelligently reducing battery consumption and privacy concerns.

[0062] (Schedule-based monitoring and control) As shown in Figure 16, the ON / OFF status of the monitoring function may be controlled based on a pre-set schedule. The caregiver A can use the GUI of the monitoring app to set the days and times when they want continuous monitoring to be enabled (for example, the weekday school commuting times of "7:30-8:30" and "15:00-17:00") in a calendar format, according to the lifestyle pattern of the person being monitored B. This setting information is stored on the server 30. The terminal control unit 110 of the monitored terminal 10 refers to its built-in clock function and turns on the continuous monitoring function only when the current time falls within the set time period, and turns it off at other times (for example, during school classes or at night when at home). This stops monitoring during times when the likelihood of encountering danger is low, enabling more efficient battery operation.

[0063] <4. Examples> The operation of the present invention will be described in more detail below with reference to specific embodiments.

[0064] (Example 1: Sudden danger while walking home from school) Taro, an 8-year-old elementary school student who is the person being monitored (Person B), was walking home from school with the monitoring device 10 attached to his schoolbag. As Taro was walking alone near a park, a stranger suddenly grabbed his arm. Terrified, Taro cried out, "Help!" The monitored terminal 10 is in constant monitoring mode because it is during dismissal time, due to its smart operation function (schedule-based control). The acoustic data acquisition unit 120 acquires acoustic data from the microphone 16, including Taro's shouts and ambient noise, and streams it to the server 30. The danger determination unit 340 of server 30 analyzes the received acoustic data. First, it detects that the volume exceeds a threshold of 90 dB (Step S211: YES). Next, it detects the keyword "help" through speech recognition processing (Step S212: YES). Furthermore, the machine learning model 341 analyzes the voice pattern and outputs a probability of "child in distress" of 0.98. Since all of these conditions are met, the danger determination unit 340 determines with extremely high accuracy that a dangerous situation has occurred (Step S214: YES).

[0065] The server 30 immediately sends a second notification to the monitoring terminals 20 of Taro's mother and father (step S202). The mother's monitoring device 20 (smartphone) was set to silent mode, but it was forcibly deactivated, and a loud warning sound was emitted at maximum volume. At the same time, the entire screen flashed red, and the text "[Emergency] Taro is in danger!" and a large map of the park showing Taro's current location were displayed (see Figure 14). Furthermore, as a warning sound, the actual scream of "Help!" that had been detected earlier was played on a loop. The mother was startled by the unusual notification and immediately picked up her smartphone. As she confirmed on the map on the screen that the location was a park, she heard her son's plea for help and the rough voice of a man who seemed to be the perpetrator through the voice call that the system had automatically established. The mother immediately grasped the seriousness of the situation and, with her other hand, called 110 (the Japanese emergency number), providing the police with the precise location information displayed on her smartphone screen and the sounds she could hear. As a result, police officers rushed to the scene and Taro was safely rescued.

[0066] (Example 2: Falls among the elderly) Ms. Suzuki (85 years old), who is the person being monitored (Person B), is spending time alone at home. She is wearing the monitoring device 10 as a pendant around her neck. When she tried to stand up in the living room, she tripped and fell, hitting her body hard on the floor. The acoustic data acquisition unit 120 of the monitored terminal 10 acquires ambient sounds in constant monitoring mode. It detects the loud impact sound (high volume of low-frequency sound) that occurred when the person fell, and the subsequent groans of Ms. Suzuki saying, "Ugh...it hurts...I can't move." The danger determination unit 340 of server 30 first detects that the frequency characteristics of the impact sound are similar to the "collision sound" pattern (Step S213: YES). Furthermore, it detects the keywords "painful" and "unable to move" from the subsequent audio (Step S212: YES). Based on this, the system determines that a dangerous situation has occurred.

[0067] Server 30 sends a second notification to the daughter's monitoring device 20, who lives separately. The daughter's smartphone displays a notification that reads "[Urgent] Something seems to have happened to your mother," along with a map showing her home location. A faint groan from her mother can be heard through the automatically established voice call. The daughter quickly understands the situation and is able to contact her family doctor and a relative living nearby to request prompt assistance.

[0068] <5. Variation> The present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.

[0069] (Variation 1: Judgment by on-device AI) In the above embodiment, an example was mainly described in which the server 30 performs risk assessment. However, with the recent advancements in edge computing technology, it is also possible for the monitored terminal 10 to perform more sophisticated assessments. The control unit 11 of the monitored terminal 10 employs a processor equipped with an NPU (Neural Processing Unit) specialized for AI processing. The machine learning model 341 trained on the server 30 is then lightweighted and quantized in a format such as TensorFlow Lite and stored in the memory unit 12 of the monitored terminal 10. The monitored terminal 10 analyzes the acoustic data acquired from the microphone 16 in real time using its own danger detection unit 150, without transmitting it to the server. Only when danger is detected does it send an emergency notification request to the server 30. This configuration dramatically reduces the amount of communication data, thereby extending battery life and lowering communication costs. Furthermore, since sensitive information such as the surrounding sounds of the monitored person is not transmitted outside the terminal, it is also excellent from a privacy protection standpoint.

[0070] (Variation 2: Gradual notification levels) The danger determination unit 340 may determine the level of danger in multiple stages and change the notification method accordingly. For example, if only a "loud noise" is heard, it will be determined to be a "Level 1" danger and the text "There was a loud noise" will be added to the normal first notification. On the other hand, if the keyword "help" is detected in addition to a "loud noise," it will be determined to be a "Level 2" danger and a second notification will be issued, including forced alarm and location information display. Furthermore, if a "scream" is detected and a "large impact" is detected by the acceleration sensor, it will be determined to be the highest "Level 3," and in addition to the second notification, it will also establish an automated voice call channel and automatically notify pre-registered police and security companies. In this way, by escalating notifications according to the detected event, a more detailed and appropriate response becomes possible.

[0071] (Modification 3: Integration with other IoT devices) The monitoring system 1 may also be linked with other IoT (Internet of Things) devices. For example, if danger is detected while the person being monitored B is at home, the server 30 sends a second notification to the monitoring terminal 20 and simultaneously sends commands via the smart home hub to flash smart lights throughout the house or to sound a loud alarm from a smart speaker. This can alert other family members inside the house to the danger or alert people outside to the anomaly and encourage third-party intervention.

[0072] (Example 4: Advanced power-saving logic) The smart operation functions can be further enhanced. For example, in distance-based monitoring and control, GPS positioning consumes a lot of battery power, making it inefficient to always use it for proximity determination with the monitoring terminal 20. Therefore, BLE (Bluetooth Low Energy) is normally used to scan the BLE radio waves of the monitoring terminal 20, and proximity (e.g., within 10m) is determined by the received signal strength (RSSI). Only when the BLE radio waves can no longer be detected is the GNSS receiver 14 activated to measure the accurate distance. In addition, by utilizing the information from the accelerometer installed in the monitored terminal 10, if the terminal is stationary for a long period of time (e.g., 30 minutes or more) (e.g., placed on a desk in a school classroom, charging at home, etc.), it is determined that the possibility of danger is low, and the constant monitoring mode is temporarily suspended. When the accelerometer detects that the terminal has started moving, monitoring is automatically resumed. By combining these multiple pieces of information, it is possible to maximize battery life. [Explanation of symbols]

[0073] 1…Monitoring system 10…Monitored device 11, 21, 31… Control Units 12, 22, 32...Storage section 13, 23, 33... Communications Department 14, 24...GNSS receiver 15, 26... Input section 16, 27... Mike 17, 28... Speakers 18... Vibrator 19... Bus 20... Monitoring device 25...Display section 30… Server 110... Terminal Control Unit 120... Acoustic data acquisition unit 130...location information acquisition unit 140...Notification Processing Unit 150... (Danger detection unit of the monitored device) 310... Server Control Unit 320... Communications Relay Unit 330...User Management Department 340... (Server) Risk Assessment Unit 341…Machine learning models 361...User mapping information A...Guardian B... Person being watched over NW...Communication Network (Note 1) (Problem) In situations where the person being monitored is involved in a sudden accident or incident and is unable to operate a device to call for help, there was a need for a means to quickly detect the danger and reliably notify the person being monitored. (Content) A method for detecting a dangerous situation based on acoustic data acquired by the microphone of a portable monitoring device, and in response to the detection, sending a second notification that is more urgent than the first notification, which is a normal notification. (Effect) Even when the person being monitored is unable to operate the system, it can autonomously detect danger from surrounding sounds and forcibly deliver a warning regardless of the monitor's settings, enabling a rapid initial response. (Note 2) (Challenge) There was a need to detect auditory signs of danger, particularly screams and collision sounds, from among a variety of dangerous situations, using a simple configuration. (Content) A dangerous situation is detected when the volume of the audio data exceeds a predetermined threshold. (Effect) By capturing changes in volume that are likely to occur when danger is present, such as shouts or loud noises, danger can be detected with simple processing. (Note 3) (Challenge) It was necessary to detect situations where the content of speech contained danger, even if the volume was not loud (e.g., a quiet plea for help). (Content) A dangerous situation is detected when the audio data contains pre-registered keywords such as "help". (Effect) By using speech recognition technology, it becomes possible to detect dangers based on linguistic meaning, which cannot be determined solely by the volume of sound. (Note 4) (Challenge) It was necessary to identify specific event sounds, such as screams and collision sounds, not just human language, in order to detect danger. (Content) A dangerous situation is detected when the acoustic data has predetermined frequency characteristics corresponding to screams or collision sounds. (Effect) Acoustic analysis allows for the identification of specific hazardous events from non-verbal sounds, enabling responses to a wider range of hazardous scenarios. (Note 5) (Challenge) There was a need for more accurate danger detection that was difficult to achieve with simple rules, such as distinguishing between a child's playful shouting and a shouting in a truly dangerous situation. (Content) This system analyzes acoustic data using a machine learning model that has learned audio patterns suggesting child distress, and detects dangerous situations. (Effect) By using machine learning, it is possible to identify the context of danger from complex acoustic patterns and achieve highly accurate danger detection while suppressing false positives. (Note 6) (Problem) The caregiver who received the second notification lacked sufficient information to immediately grasp the specific circumstances of the danger and determine the optimal course of action. (Content) In response to the second notification, a two-way voice communication channel is automatically established between the monitored device and the monitoring device without any additional user action. (Effects) Monitors can immediately hear firsthand accounts from the scene, dramatically improving the speed and accuracy of situation assessment and enabling more appropriate initial responses. (Note 7) (Problem) When a danger occurs, if the specific location is unknown, it is difficult for caretakers to rush to the scene or report it to the police, etc. (Content) Emergency notifications will include location information indicating the current location of the monitored device. (Effect) It allows for the immediate identification of the location of a danger, facilitating the rapid arrival of caretakers and rescue organizations at the scene. (Note 8) (Problem) There was a possibility that the urgent second notification might be overlooked amidst routine notifications. (Content) The second notice will be displayed in a more visually emphasized manner than a normal notice (e.g., warning colors, larger font). (Effects) The caregiver can recognize an emergency at a glance, prevent notifications from being overlooked, and encourage immediate action. (Note 9) (Problem) There was a risk that the caregiver might miss a notification even if it was sent only once while they were temporarily away from the device. (Content) The device will continue to emit an audible sound or vibrate until the caregiver performs a confirmation operation. (Effect) By providing persistent notifications, caregivers can ensure that an emergency has occurred, even at a later date, thus improving the reliability of the system. (Note 10) (Problem) General-purpose warning sounds have the problem of not being able to intuitively convey the urgency or type of danger. (Content) As an emergency notification warning sound, the system plays the actual sound data (such as actual shouts) used for danger detection. (Effect) By letting the observer hear the actual sounds, the seriousness and situation of the situation can be conveyed instantly and intuitively, prompting a quick decision. (Note 11) (Problem) When the system is set up so that only one primary caregiver receives notifications, there is a risk that danger may be overlooked if that person is unable to respond. (Content) The second notification will be sent simultaneously to all associated monitoring devices, regardless of the normal notification settings. (Effect) By sharing danger information among multiple observers, the likelihood of someone being able to respond increases, improving the certainty of rescue. (Note 12) (Challenges) Continuous voice monitoring consumes a lot of power for battery-powered mobile devices, is unnecessary when a caregiver is nearby, and raises privacy concerns. (Content) Acoustic data acquisition (monitoring) is performed only when the distance between the monitored device and the monitoring device is greater than a predetermined distance. (Effect) By automatically stopping unnecessary monitoring, battery consumption can be reduced and privacy can be protected. (Note 13) (Problem) Continuously monitoring users uniformly, regardless of their lifestyle patterns, was inefficient from a battery consumption standpoint. (Content) Acquisition (monitoring) of acoustic data will be performed only during pre-set days of the week and time periods (e.g., during commuting to and from school). (Effect) By focusing monitoring on time periods with a high risk of danger, battery usage can be made more efficient, and user convenience can be improved. (Note 14) (Challenge) It was necessary to reliably detect dangerous sounds while the person being monitored was in the process of recording a voice message asking for help. (Content) Acquisition of audio data is performed while recording voice messages in response to user input. (Effect) By including dangers that occur during the active actions of the person being monitored as detection targets, the detection opportunities are expanded and the system's comprehensiveness is enhanced. (Note 15) (Problem) In situations where the person being monitored is involved in a sudden accident or incident and is unable to operate a device to call for help, there was a need for a means to quickly detect the danger and reliably notify the person being monitored. (Contents) A monitoring system including a monitored terminal, a monitoring terminal, and a server, configured to implement the method described in Appendix 1. (Effect) The technical concept of the present invention can be protected as a concrete "thing" system. (Note 16) (Problem) There was a need to provide specific means for realizing the monitoring system of the present invention on a computer. (Contents) A program to enable a computer to function as a server or monitored terminal of the monitoring system described in Appendix 15. (Effect) The technical concept of this invention can be protected in the form of software, and acts such as its manufacture, distribution, and transmission can be included within the scope of the rights.

Claims

1. A method for a monitoring system to be performed, which includes a portable monitoring terminal equipped with a battery, a monitoring terminal, and a server, The monitoring terminal acquires ambient acoustic data using its microphone, The steps include: the monitoring terminal or the server detecting a predetermined danger state based on the acoustic data, which indicates that there is a possibility that the person being monitored is in danger; In response to the detection of the aforementioned dangerous condition, the server sends a second notification, which is more urgent than a first notification (a normal notification), to the monitoring terminal. A monitoring method characterized by including [this].

2. The monitoring method according to claim 1, characterized in that the detection of the dangerous state is performed when the volume of the sound data exceeds a predetermined threshold.

3. The monitoring method according to claim 1, characterized in that the detection of the dangerous state is performed when the acoustic data includes a pre-registered keyword such as "help".

4. The monitoring method according to claim 1, characterized in that the detection of the dangerous condition is performed when the acoustic data has predetermined frequency characteristics corresponding to a scream or collision sound.

5. The monitoring method according to claim 1, characterized in that the detection of the dangerous situation is performed by analyzing the acoustic data using a machine learning model that has learned voice patterns suggesting a child is lost.

6. The monitoring method according to claim 1, further comprising the step of automatically establishing a bidirectional voice communication path between the monitored terminal and the monitoring terminal without requiring any additional user operation, in response to the transmission of the second notification.

7. The monitoring method according to claim 1, characterized in that the second notification includes location information indicating the current location of the monitored terminal.

8. The monitoring method according to claim 1, characterized in that the second notification is displayed on the monitoring terminal in a manner that is more visually emphasized than the first notification.

9. The monitoring method according to claim 1, characterized in that the second notification continues to emit an audible sound or vibrate in the monitoring terminal until the monitoring person performs a confirmation operation.

10. The monitoring method according to claim 1, characterized in that the audible sound to be emitted in the second notification is the acoustic data or a part thereof used to detect the dangerous condition.

11. The monitoring method according to claim 1, characterized in that, if there are multiple monitoring terminals associated with the monitored terminal, the second notification is sent to all of the multiple monitoring terminals, regardless of whether or not a setting has been made to restrict the recipients of the first notification.

12. The monitoring method according to claim 1, characterized in that the step of acquiring the acoustic data is performed when the distance between the monitored terminal and the monitoring terminal is greater than a predetermined distance.

13. The monitoring method according to claim 1, characterized in that the step of acquiring the aforementioned acoustic data is performed on a predetermined day of the week or time period.

14. The monitoring method according to claim 1, characterized in that the step of acquiring the acoustic data is initiated in response to an operation on the input unit of the monitored terminal and is performed during the recording of an audio message.

15. A monitoring system including a portable monitoring terminal equipped with a battery, a monitoring terminal, and a server, The aforementioned monitoring terminal is equipped with a microphone, The monitored terminal or the server detects a dangerous situation based on the acoustic data acquired by the microphone. In response to the detection of the aforementioned dangerous condition, the server is configured to send a second notification, which is more urgent than the first notification (a normal notification), to the monitoring terminal. A monitoring system characterized by the following features.

16. A program for causing a processor to function as the server or the monitored terminal in the monitoring system described in claim 15.

Images