system

Abstract

We provide the system. [Solution] A means for receiving video data acquired from an information acquisition device in real time, A means for converting received video data into an analyzable format, An analysis means for detecting abnormal behavior based on converted video data, A means of notifying relevant authorities when abnormal behavior is detected, Means of notifying relevant parties of an anomaly, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to ensure the safety of the elderly and infants, round-the-clock monitoring is required, but there are limitations to manual monitoring. In addition, systems using multiple sensors and specialized devices are often costly and impose a large economic burden. Therefore, there is a need for a system that can efficiently and economically detect abnormal situations in the living environment and automatically respond quickly.

Means for Solving the Problems

[0005] This invention includes an analysis means for detecting abnormal behavior by receiving video data in real time from an information acquisition device and converting the data into an analyzable format. Furthermore, it includes means for automatically notifying relevant organizations and immediately notifying relevant parties when abnormal behavior is detected. This makes it possible to ensure the safety of the elderly and infants economically and efficiently.

[0006] An "information acquisition device" is a device, such as a surveillance camera or sensor device, that acquires the surrounding environment and the actions of an object as image data.

[0007] "Video data" refers to data that represents visual information acquired by an information acquisition device in digital format.

[0008] "Real-time" refers to processing or responding to an event almost simultaneously with its occurrence, resulting in minimal time delay.

[0009] An "analyzable format" is data that has been converted into a format that a computer can recognize and process.

[0010] "Abnormal behavior" refers to actions or states that deviate from normal behavioral patterns and may be a sign of unexpected accidents or dangers.

[0011] "Analysis means" refers to algorithms and processes that use digital data to analyze the behavior and situation of an object and to determine specific patterns or anomalies.

[0012] A "notification system" refers to a mechanism or method for transmitting information to pre-designated relevant organizations or individuals when an abnormal situation is detected.

[0013] "Notification means" refers to methods or processes for informing relevant parties about the occurrence of an anomaly or the status of the system. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a system that detects abnormal behavior by receiving video data in real time from information acquisition devices such as surveillance cameras and analyzing that video data. The system consists of three components: a server, a terminal, and a user, each of which is responsible for a specific function.

[0036] The server continuously receives video data from the information acquisition device. The received data is sent to a terminal, where it is converted into an analyzable format. The terminal uses a specific algorithm to analyze the video data and detect abnormal behavior. For example, falls and prolonged periods of remaining stationary in the same position are among the behaviors that can be detected.

[0037] If abnormal behavior is detected, the device immediately initiates a notification process to the relevant authorities. This notification includes the location and timing of the abnormal behavior, and, if necessary, a portion of the video footage. Notifying relevant parties allows for quick sharing of the situation and enables a swift response.

[0038] Users can remotely monitor the status of the monitoring system using a mobile application. When an anomaly is detected, an alert is sent, allowing for immediate on-site verification and necessary action. For example, even if a user is in a remote location, they can view on-site video through the app and receive support to respond appropriately.

[0039] As a concrete example, consider a case where an elderly person living alone falls in their living room. When a surveillance camera detects the anomaly, the data is analyzed on a terminal via a server, confirming the fall. The terminal immediately notifies emergency services and sends an alert to registered family members, prompting a quick response. This system makes it possible to more reliably monitor the safety of the elderly and infants and to respond quickly in emergencies.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server receives video data in real time from the information acquisition device. This provides continuous visual information within the monitoring area.

[0043] Step 2:

[0044] The server converts the received video data into an analyzable format. Pre-processing, such as image resolution adjustment and noise reduction, is performed during this stage.

[0045] Step 3:

[0046] The device acquires the converted video data and uses a generating AI to analyze abnormal behavior. The AI ​​compares it to learned normal behavior patterns to determine if there are any deviations.

[0047] Step 4:

[0048] If the terminal determines, based on the analysis results, that abnormal behavior has occurred, it will save the data and activate the means to notify the appropriate authorities. Specifically, an emergency notification message will be created and sent.

[0049] Step 5:

[0050] The device initiates the notification process to relevant parties. Users receive an alert via the mobile app, which includes a detailed notification of the situation.

[0051] Step 6:

[0052] Based on alerts received by the user, on-site video footage can be viewed through the app. This allows the user to respond quickly and send additional instructions to relevant parties as needed.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] For the elderly and individuals requiring specific risk management, there is a need for more efficient and rapid response monitoring systems that provide safety and security. However, current technology lacks the ability to detect abnormal behavior in real time and immediately notify the appropriate organizations and individuals. Furthermore, analyzing the data requires multiple processing steps, and the lack of system integration to smoothly carry out these processes is a challenge.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server is a device that receives video data acquired from an information acquisition device in real time and transmits it to a data processing device; a device that converts the received video data into an analyzable data format to enable analysis in the next stage; and an analysis device that detects abnormal behavior from the converted data using a specific algorithm. This makes it possible to quickly detect abnormal behavior and immediately notify relevant organizations and individuals.

[0058] An "information acquisition device" is a hardware device used to acquire video data from a monitored object.

[0059] A "data processing device" is a device that converts received data into an analyzable format and then performs further analysis on it.

[0060] An "analysis device" is a device used to detect abnormal behavior from converted video data, and it performs analysis using a specific algorithm.

[0061] A "communication device" is a device used to notify relevant organizations of information when abnormal behavior is detected.

[0062] A "notification device" is a device that notifies registered parties of information when an anomaly is detected.

[0063] A "specific algorithm" is a computational method used to detect abnormal behavior from data.

[0064] "Data format" refers to the data structure and encoding necessary for analyzing video data.

[0065] This invention is a safety monitoring system for the elderly and individuals requiring specific risk management. The system consists of three components: a server, a terminal, and a user.

[0066] The server receives video data in real time from information acquisition devices such as surveillance cameras. This data is first transmitted to a data processing unit. The server is equipped with high-performance communication equipment to ensure stable data reception.

[0067] The terminal converts the video data sent from the server into an analyzable data format. Here, it compresses the data using a specific codec and adjusts the resolution as needed. In particular, it runs a specific algorithm to analyze abnormal behavior and detect it rapidly. The algorithm used by the analysis device extracts abnormal patterns based on stored training data and recognizes anomalies in real time.

[0068] When abnormal behavior is detected, the device notifies the relevant authorities via a communication device. At the same time, it also notifies registered contacts to promptly take necessary action. This enables a smooth response in emergencies, such as when an elderly person falls.

[0069] Users can remotely monitor the system's operational status using a mobile application. The application immediately displays an alert when an anomaly is detected, allowing users to understand the situation on-site. Users can also request additional information from the app as needed.

[0070] An example of a prompt is, "Please describe the fall detection system for the elderly. This system analyzes video data in real time and detects abnormal behavior." Using this prompt allows for efficient communication of the system's purpose and usage conditions to the generated AI model.

[0071] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0072] Step 1:

[0073] The server receives video data in real time from the information acquisition device. The input is raw data from surveillance cameras, and the output generates stream data for transmission to the data processing device. Specifically, it integrates data from multiple cameras and utilizes a communication protocol to achieve consistent data distribution.

[0074] Step 2:

[0075] The terminal receives raw stream data sent from the server and converts it into a parseable format. The input is stream data from the server, and the output is data in a format suitable for the analysis algorithm. Specifically, it performs data compression using a codec and adjusts the resolution as needed, then sends the converted data to the next processing step.

[0076] Step 3:

[0077] The device analyzes the converted data using a specific algorithm to detect abnormal behavior. The input is data in a parsable format, and the output is the result of detecting abnormal behavior. The specific operation involves using a machine learning model to recognize anomalies in real time based on training data. Based on this analysis, the type and severity of the anomaly are identified.

[0078] Step 4:

[0079] The terminal immediately notifies the relevant authorities using its communication device upon detecting abnormal behavior. The input is the result of the detected abnormal behavior, and the output is the notification message to the relevant authorities. Specific actions include transmitting information such as the location and time the abnormality occurred, and, if necessary, attaching a portion of the video footage.

[0080] Step 5:

[0081] Users receive anomaly notifications via a mobile application and check the situation on-site. Input is anomaly notifications from the device, and output is an alert displayed to the user. Specifically, users can use the app to view detailed video footage and issue instructions or additional notifications to the site as needed.

[0082] (Application Example 1)

[0083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In recent years, with the increasing importance of security, there has been a growing demand for monitoring systems to detect abnormal behavior. However, many conventional systems lack sufficient remote means to immediately confirm the situation on-site and respond, resulting in a lack of rapid and appropriate response. In particular, the difficulty for users to quickly grasp the situation and respond via mobile devices can lead to security vulnerabilities.

[0085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0086] In this invention, the server includes means for receiving video information acquired from an information acquisition means in real time, means for converting the received video information into an analyzable information format, analysis means for detecting abnormal behavior based on the converted video information, and means for notifying a mobile information terminal when abnormal behavior is detected. This enables the user to immediately detect anomalies through the monitoring system, confirm the situation via a mobile information terminal, and respond quickly.

[0087] "Information acquisition means" is a general term for devices and processes that acquire video information from the external environment.

[0088] "Means of receiving in real time" refers to technologies or methods for continuously incorporating video information into a system without delay.

[0089] An "analyzable information format" is a structured data format that a system needs to understand and process.

[0090] "Analysis means" refers to a process that includes algorithms and processing methods for analyzing converted video information and identifying abnormal behavior.

[0091] "Abnormal behavior" refers to actions that deviate from normal or expected behavioral patterns and carry identified risks within a system.

[0092] "Means of notifying relevant agencies" refers to methods or techniques for notifying pre-designated agencies of the situation when an anomaly is detected.

[0093] "Means of notifying of anomalies" refers to the function of a system that provides information to relevant parties when unusual behavior is detected.

[0094] A "portable information terminal" is a portable information processing device used by users to check system status and respond remotely.

[0095] To realize this invention, three parties—the server, the terminal, and the user—must each fulfill their respective roles.

[0096] The server receives video information in real time from information acquisition devices. Specifically, it collects data directly from surveillance cameras and sensors, converts it into a processable format, and transmits it to the terminal. Video streaming technology and data transfer protocols are used for this process.

[0097] The terminal analyzes video information received from the server to detect abnormal behavior. The analysis uses an algorithm that detects deviations from normal behavior patterns based on training data. Image processing libraries such as OpenCV are used in this analysis to identify situations such as illegal entry, fires, and falls.

[0098] When abnormal behavior is detected, the device immediately notifies the relevant authorities and sends a notification to the user via the mobile device. This notification includes images of the scene and a description of the situation, allowing the user to immediately check the situation and take appropriate action.

[0099] Users can remotely check the status of the monitoring system using a mobile device. When an anomaly is detected, an alert is immediately sent, allowing users to quickly understand the situation on-site and take necessary responses or notifications. An example of a prompt message generated using AI is: "Generate a scenario in which a surveillance camera installed in an office detects an illegal intrusion at night. Include the system notification content and the administrator's response flow."

[0100] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0101] Step 1:

[0102] The server receives video information in real time from the information acquisition means. The input is video data from surveillance cameras, and the server performs initial processing to send it to the terminal in a predetermined data format. The output is video data converted into an analyzable format. The server receives data stably over the network and performs compression and encoding to enable reliable data transfer.

[0103] Step 2:

[0104] The terminal prepares the video data received from the server into an analyzable format. The input is the video data sent from the server, which is decompressed and formatted as needed. The output is a dataset in a format usable by the analysis engine. Here, for example, consecutive frames are extracted from a JPEG image and prepared for analysis.

[0105] Step 3:

[0106] The device analyzes abnormal behavior using a prepared dataset. The input is transformed sequential frame data, and the output is the identification and location information of the detected abnormal behavior. Data processing is performed by running a motion detection algorithm using OpenCV. The device evaluates the differences in movement between frames and the length of time spent stationary, and detects deviations from the set normal behavior pattern.

[0107] Step 4:

[0108] When an anomaly is detected, the device initiates the process of notifying the relevant authorities. The input is the result of the detected abnormal behavior, and the output is the report content and a video clip. The device automatically compiles the necessary information and transmits it via email or API. Specifically, it generates a notification that includes a portion of the video, the time of occurrence, and location information.

[0109] Step 5:

[0110] The terminal notifies the mobile device of detected anomalies. Inputs are the detection results and video clips of the abnormal behavior, while output is an alert sent to the user. The terminal uses a communication protocol to send push notifications to the mobile device, allowing the user to immediately check the situation. As a concrete example, a notification appears on the user's smartphone, and they can view the on-site video through the application.

[0111] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0112] This invention relates to a system that uses video and audio data obtained from information acquisition devices such as surveillance cameras and microphones to monitor the safety of the elderly and infants in real time. This system incorporates an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection.

[0113] This system begins with a server collecting video and audio data received from an information acquisition device. The server then transmits this data to a terminal, where it is formatted into an analyzable format. The generating AI on the terminal uses the video data to detect abnormal behavior and simultaneously uses an emotion engine to analyze the user's emotions from the audio data.

[0114] The emotion engine uses emotion recognition technology to identify a user's emotional state from their tone of voice and speaking style. For example, it evaluates emotions such as whether an elderly person is feeling anxious or whether an infant is showing signs of sudden crying, and comprehensively detects signs of abnormality. If an abnormality is detected, the device uses that data to make an emergency call to the relevant authorities. If necessary, parts of the relevant video or audio will be attached to the call.

[0115] Users are immediately notified of the results of anomaly detection and emotion recognition. Through the mobile app, users can monitor the situation on-site and the evolution of emotions in real time. This feature enables faster and more appropriate responses, allowing for additional instructions and adjustments as needed.

[0116] As a concrete example, consider a scenario where an elderly person living alone suddenly expresses anxiety. In this case, the terminal analyzes video and audio data and determines that the person is experiencing anxiety. Based on this, it can send an alert to the appropriate contacts and, if necessary, dispatch medical staff. This system enables advanced anomaly detection that incorporates changes in emotions, providing a support system that allows people to live their daily lives with greater peace of mind.

[0117] The following describes the processing flow.

[0118] Step 1:

[0119] The server receives video and audio data in real time from the information acquisition device. This allows for the acquisition of visual and auditory information about the monitored target.

[0120] Step 2:

[0121] The server sends the received data to the terminal. The terminal converts this data into a parseable format and prepares for the next processing step.

[0122] Step 3:

[0123] The terminal uses a generating AI to analyze abnormal behavior based on the converted video data. The AI ​​compares the current behavior with past behavioral patterns to detect unusual actions and movements.

[0124] Step 4:

[0125] The device uses an emotion engine to analyze the user's emotions based on voice data. It analyzes the pitch and tone of the voice to evaluate emotions such as anxiety and stress.

[0126] Step 5:

[0127] The device integrates video and emotion analysis results to make a comprehensive judgment about the presence or absence of anomalies. If an anomaly is detected, it triggers the next action.

[0128] Step 6:

[0129] The terminal uses anomaly detection data to send an emergency alert to nearby relevant organizations. In addition to conventional messages, the alert can also include analyzed emotional information.

[0130] Step 7:

[0131] The device instantly sends notifications of anomaly detection and emotion recognition to the user via a mobile app. This allows the user to immediately check the situation and take action.

[0132] Step 8:

[0133] The user receives a notification, opens the app, and checks the video and audio from the scene. This allows the user to understand the situation and send out necessary instructions.

[0134] (Example 2)

[0135] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0136] In modern times, effectively monitoring the safety of the elderly and infants is a critical challenge. Traditional monitoring systems have limitations in detecting abnormal behavior and require further improvement. Furthermore, they cannot capture emotional changes, potentially missing signs of abnormality. Overcoming these challenges and providing a safer living environment is essential.

[0137] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0138] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, means for detecting abnormal behavior using a generated AI model based on the converted data, means for analyzing emotional states from audio data based on emotion recognition technology, means for notifying relevant organizations when abnormal behavior or anxiety is detected, and means for notifying relevant parties of the abnormality and allowing them to check the situation in real time on a mobile terminal. This enables comprehensive safety management that takes into account not only the detection of abnormal behavior but also changes in emotions.

[0139] An "information acquisition device" is a device that collects data such as video and audio and provides it to a server.

[0140] "Real-time" refers to a state where information is processed and analyzed as soon as it is generated, and the results are reflected immediately.

[0141] An "analyzable format" refers to a format in which collected data has been appropriately processed and can be analyzed using AI models or other tools.

[0142] A "generative AI model" is an artificial intelligence model that recognizes patterns from new data based on previously learned data and performs specific tasks.

[0143] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns and is an important indicator that should be monitored.

[0144] "Emotion recognition technology" refers to technology that identifies a person's emotional state based on voice and text data.

[0145] "Reporting" refers to the act of informing the relevant authorities about information regarding detected anomalies.

[0146] A "mobile device" is a device that a user can carry with them and is used to receive information and perform operations via applications.

[0147] "Related organizations" refers to public or private organizations that provide support or response when an anomaly occurs.

[0148] Embodiments of this invention include the following systems and functions.

[0149] The system is initially composed of a server that receives video and audio data in real time using information acquisition devices such as surveillance cameras and microphones. The server receives the data via a communication interface and immediately transmits it to terminals. The terminals use a data conversion module to process the received data into a format that can be analyzed.

[0150] The device is equipped with a generative AI model for analyzing video data, enabling the detection of abnormal behavior. Furthermore, emotion recognition technology is used for analyzing audio data. This technology analyzes the tone and speed of speech to identify the user's emotional state. This enables advanced safety management that captures not only abnormal movements but also emotional changes.

[0151] If an anomaly is detected, the device automatically notifies the relevant authorities. The notification includes data on the detected abnormal behavior and emotional changes. Furthermore, the user is immediately notified of the anomaly, and this information can be viewed on their mobile device. In this way, the user can understand the situation in real time and take necessary actions quickly.

[0152] As a concrete example, in a living environment with an elderly person living alone, this system could detect an increase in the elderly person's anxiety and promptly notify care services or medical institutions. This would allow the user to live their daily life with greater peace of mind. An example of a prompt message would be, "Detect anxiety from the elderly person's tone of voice and suggest appropriate measures."

[0153] Ultimately, this system aims to leverage its advanced data processing capabilities to support users' lives while ensuring safety.

[0154] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0155] Step 1:

[0156] The server receives video and audio data in real time from surveillance cameras and microphones. The input data consists of stream data from the cameras and microphones, which is acquired by a network module within the server. The output is the collected raw video and audio data. Specifically, the server uses a communication protocol to ensure uninterrupted data reception.

[0157] Step 2:

[0158] The server sends the received data to the terminal. The input is the raw data mentioned earlier, and a communication format conversion is performed to convert it into a format suitable for transmission. The output is data packets grouped into an appropriate batch size. In actual operation, the server uses its data transmission function to ensure stable data transfer.

[0159] Step 3:

[0160] The terminal formats the received data into a format that can be analyzed. The input is data packets sent from the server, which are processed by the data conversion module. The output is data formatted for analysis, such as video frames and audio clips. Specifically, the terminal performs frame extraction and noise reduction before analyzing the data.

[0161] Step 4:

[0162] The on-device AI detects abnormal behavior. The input is formatted video data, and the AI ​​model is executed. The output is data of the detected abnormal behavior. Specifically, the AI ​​model analyzes the movement pattern for each frame and captures movements that deviate from normal movement patterns.

[0163] Step 5:

[0164] The device uses an emotion engine to analyze emotional states using voice data. The input is formatted voice data to which emotion recognition technology is applied. The output is information on the analyzed emotional state. Specifically, the model extracts embedded features such as voice tone and speech rate to estimate the emotional state.

[0165] Step 6:

[0166] The device will notify the relevant authorities if it detects abnormal behavior or anxiety. The input is the data on abnormalities and emotions obtained in the previous step, and a notification message is generated based on this data. The output is an emergency alert sent to the designated contact. Specifically, the message generation program attaches the relevant data to ensure rapid notification.

[0167] Step 7:

[0168] Users receive anomaly notifications and check the real-time status on their mobile devices. The input is notification data sent from the device, which allows for understanding the current situation. The output is a notification display and an interactive interface that the user can view and operate. Specifically, the mobile app displays notifications as pop-ups and provides functions to visualize video and sentiment analysis results.

[0169] (Application Example 2)

[0170] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0171] To ensure the safety of the elderly and infants in real time, it is necessary not only to detect abnormal behavior but also to quickly capture changes in their emotions. However, conventional technology is limited to detecting abnormal behavior and has the drawback of not being able to detect early signs of abnormality through emotion analysis from voice.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0173] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, and means for detecting abnormal behavior from the converted data and analyzing emotional states from the audio data. This enables rapid and appropriate responses through highly accurate anomaly detection and emotion analysis.

[0174] An "information acquisition device" is a device that has the function of collecting video data and audio data.

[0175] "Video data" refers to image information acquired by surveillance cameras and other video devices.

[0176] "Audio data" refers to sound information acquired by a microphone or similar device.

[0177] A "means of receiving data in real time" refers to a mechanism that processes acquired data almost simultaneously.

[0178] "Means of converting to an analyzable format" refers to a function that processes received data into a format that can be processed by an analysis engine.

[0179] An "analysis means for detecting abnormal behavior" is a mechanism that analyzes video data to identify deviations from normal behavior.

[0180] A "means for analyzing emotional states" refers to a mechanism that analyzes voice data to identify the user's emotions.

[0181] "A means of notifying mobile devices and reporting to relevant organizations" refers to a system that transmits information to portable communication devices or relevant departments when an anomaly is detected.

[0182] "Means for transmitting video and audio of abnormal behavior to relevant organizations" refers to a function that transmits data that serves as evidence of detected abnormalities to public or designated organizations.

[0183] The system necessary to implement this invention consists of an information acquisition device, a server, a terminal, and a mobile device used by the user. The server receives video and audio data acquired from surveillance cameras and microphones in real time. The received data is converted into an analyzable format, and analysis is performed on the server to detect abnormal behavior. Video analysis libraries such as OpenCV are used for this analysis.

[0184] Furthermore, voice analysis technology is applied to the analysis of voice data to identify the user's emotional state from their tone of voice and speaking style. For example, by using voice recognition services such as Azure Cognitive Services, changes in emotion can be analyzed in real time. This combination of emotion recognition and abnormal behavior detection enables highly accurate safety monitoring.

[0185] The user's device is immediately notified of any detected anomalies. Push notifications are sent via Firebase Cloud Messaging, allowing the user to take appropriate action based on the notification. If necessary, the relevant authorities will also be notified. In such cases, video and audio recordings capturing abnormal behavior or emotional changes may be attached.

[0186] For example, if an elderly person begins to feel anxious at home, the system analyzes their emotions from their voice and detects it as an abnormality. This immediately sends a notification to the family member, allowing for a quick response while viewing the video and audio. This system provides a safe and secure monitoring environment.

[0187] An example of a prompt for using a generative AI model to more precisely analyze abnormal behavior would be: "Describe a system that monitors the daily activities of elderly people, detects emotional changes, and provides real-time notifications." This allows the AI ​​model to perform a detailed analysis and enable flexible responses.

[0188] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0189] Step 1:

[0190] The server receives video and audio data in real time from surveillance cameras and microphones. The input is raw data from the cameras and microphones, while the output is data temporarily stored within the server. This data is processed as a stream to maintain real-time performance.

[0191] Step 2:

[0192] The server converts the received raw data into a format that can be analyzed. The input is raw video and audio data, and the output is data in a format that the analysis engine can process. The OpenCV library is used to convert the video data to a standard image format, and the audio is converted to WAV or MP3 format.

[0193] Step 3:

[0194] The server detects abnormal behavior based on the converted video data. The input is the converted video data, and the output is the evaluation result regarding the abnormal behavior. OpenCV is used to analyze the movement and compare it with predetermined abnormal behavior patterns.

[0195] Step 4:

[0196] The server analyzes audio data to identify emotional states. The input is the converted audio data, and the output is the detected emotion label. Speech recognition technologies such as Azure Cognitive Services are used to analyze emotions from voice tone and speed.

[0197] Step 5:

[0198] The server determines whether an anomaly has occurred based on the detection results and outputs the result. The input is the evaluation result of abnormal behavior and the result of sentiment analysis, and the output is a flag indicating the presence or absence of an anomaly and its detailed information. If an anomaly is detected, the server prepares to send a notification request to the mobile device.

[0199] Step 6:

[0200] The server pushes information about the anomaly to the user's mobile device. The input is detailed information about the anomaly, and the output is a notification message to the user. Firebase Cloud Messaging is used for rapid notification.

[0201] Step 7:

[0202] Users receive notifications and view detailed video and audio via their mobile devices. The input is the notification message sent from the server, and the output is the video and audio content of the anomaly displayed on the device. This allows for rapid countermeasures to be taken.

[0203] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0204] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0205] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0206] [Second Embodiment]

[0207] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0208] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0209] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0210] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0211] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0212] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0213] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0214] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0215] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0216] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0217] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0218] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0219] This invention relates to a system that detects abnormal behavior by receiving video data in real time from information acquisition devices such as surveillance cameras and analyzing that video data. The system consists of three components: a server, a terminal, and a user, each of which is responsible for a specific function.

[0220] The server continuously receives video data from the information acquisition device. The received data is sent to a terminal, where it is converted into an analyzable format. The terminal uses a specific algorithm to analyze the video data and detect abnormal behavior. For example, falls and prolonged periods of remaining stationary in the same position are among the behaviors that can be detected.

[0221] If abnormal behavior is detected, the device immediately initiates a notification process to the relevant authorities. This notification includes the location and timing of the abnormal behavior, and, if necessary, a portion of the video footage. Notifying relevant parties allows for quick sharing of the situation and enables a swift response.

[0222] Users can remotely monitor the status of the monitoring system using a mobile application. When an anomaly is detected, an alert is sent, allowing for immediate on-site verification and necessary action. For example, even if a user is in a remote location, they can view on-site video through the app and receive support to respond appropriately.

[0223] As a concrete example, consider a case where an elderly person living alone falls in their living room. When a surveillance camera detects the anomaly, the data is analyzed on a terminal via a server, confirming the fall. The terminal immediately notifies emergency services and sends an alert to registered family members, prompting a quick response. This system makes it possible to more reliably monitor the safety of the elderly and infants and to respond quickly in emergencies.

[0224] The following describes the processing flow.

[0225] Step 1:

[0226] The server receives video data in real time from the information acquisition device. This provides continuous visual information within the monitoring area.

[0227] Step 2:

[0228] The server converts the received video data into an analyzable format. Pre-processing, such as image resolution adjustment and noise reduction, is performed during this stage.

[0229] Step 3:

[0230] The device acquires the converted video data and uses a generating AI to analyze abnormal behavior. The AI ​​compares it to learned normal behavior patterns to determine if there are any deviations.

[0231] Step 4:

[0232] If the terminal determines, based on the analysis results, that abnormal behavior has occurred, it will save the data and activate the means to notify the appropriate authorities. Specifically, an emergency notification message will be created and sent.

[0233] Step 5:

[0234] The device initiates the notification process to relevant parties. Users receive an alert via the mobile app, which includes a detailed notification of the situation.

[0235] Step 6:

[0236] Based on alerts received by the user, on-site video footage can be viewed through the app. This allows the user to respond quickly and send additional instructions to relevant parties as needed.

[0237] (Example 1)

[0238] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0239] For the elderly and individuals requiring specific risk management, there is a need for more efficient and rapid response monitoring systems that provide safety and security. However, current technology lacks the ability to detect abnormal behavior in real time and immediately notify the appropriate organizations and individuals. Furthermore, analyzing the data requires multiple processing steps, and the lack of system integration to smoothly carry out these processes is a challenge.

[0240] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0241] In this invention, the server is a device that receives video data acquired from an information acquisition device in real time and transmits it to a data processing device; a device that converts the received video data into an analyzable data format to enable analysis in the next stage; and an analysis device that detects abnormal behavior from the converted data using a specific algorithm. This makes it possible to quickly detect abnormal behavior and immediately notify relevant organizations and individuals.

[0242] An "information acquisition device" is a hardware device used to acquire video data from a monitored object.

[0243] A "data processing device" is a device that converts received data into an analyzable format and then performs further analysis on it.

[0244] An "analysis device" is a device used to detect abnormal behavior from converted video data, and it performs analysis using a specific algorithm.

[0245] A "communication device" is a device used to notify relevant organizations of information when abnormal behavior is detected.

[0246] A "notification device" is a device that notifies registered parties of information when an anomaly is detected.

[0247] A "specific algorithm" is a computational method used to detect abnormal behavior from data.

[0248] "Data format" refers to the data structure and encoding necessary for analyzing video data.

[0249] This invention is a safety monitoring system for the elderly and individuals requiring specific risk management. The system consists of three components: a server, a terminal, and a user.

[0250] The server receives video data in real time from information acquisition devices such as surveillance cameras. This data is first transmitted to a data processing unit. The server is equipped with high-performance communication equipment to ensure stable data reception.

[0251] The terminal converts the video data sent from the server into an analyzable data format. Here, it compresses the data using a specific codec and adjusts the resolution as needed. In particular, it runs a specific algorithm to analyze abnormal behavior and detect it rapidly. The algorithm used by the analysis device extracts abnormal patterns based on stored training data and recognizes anomalies in real time.

[0252] When abnormal behavior is detected, the device notifies the relevant authorities via a communication device. At the same time, it also notifies registered contacts to promptly take necessary action. This enables a smooth response in emergencies, such as when an elderly person falls.

[0253] Users can remotely monitor the system's operational status using a mobile application. The application immediately displays an alert when an anomaly is detected, allowing users to understand the situation on-site. Users can also request additional information from the app as needed.

[0254] An example of a prompt is, "Please describe the fall detection system for the elderly. This system analyzes video data in real time and detects abnormal behavior." Using this prompt allows for efficient communication of the system's purpose and usage conditions to the generated AI model.

[0255] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0256] Step 1:

[0257] The server receives video data in real time from the information acquisition device. The input is raw data from surveillance cameras, and the output generates stream data for transmission to the data processing device. Specifically, it integrates data from multiple cameras and utilizes a communication protocol to achieve consistent data distribution.

[0258] Step 2:

[0259] The terminal receives raw stream data sent from the server and converts it into a parseable format. The input is stream data from the server, and the output is data in a format suitable for the analysis algorithm. Specifically, it performs data compression using a codec and adjusts the resolution as needed, then sends the converted data to the next processing step.

[0260] Step 3:

[0261] The device analyzes the converted data using a specific algorithm to detect abnormal behavior. The input is data in a parsable format, and the output is the result of detecting abnormal behavior. The specific operation involves using a machine learning model to recognize anomalies in real time based on training data. Based on this analysis, the type and severity of the anomaly are identified.

[0262] Step 4:

[0263] The terminal immediately notifies the relevant authorities using its communication device upon detecting abnormal behavior. The input is the result of the detected abnormal behavior, and the output is the notification message to the relevant authorities. Specific actions include transmitting information such as the location and time the abnormality occurred, and, if necessary, attaching a portion of the video footage.

[0264] Step 5:

[0265] Users receive anomaly notifications via a mobile application and check the situation on-site. Input is anomaly notifications from the device, and output is an alert displayed to the user. Specifically, users can use the app to view detailed video footage and issue instructions or additional notifications to the site as needed.

[0266] (Application Example 1)

[0267] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0268] In recent years, with the increasing importance of security, there has been a growing demand for monitoring systems to detect abnormal behavior. However, many conventional systems lack sufficient remote means to immediately confirm the situation on-site and respond, resulting in a lack of rapid and appropriate response. In particular, the difficulty for users to quickly grasp the situation and respond via mobile devices can lead to security vulnerabilities.

[0269] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0270] In this invention, the server includes means for receiving video information acquired from an information acquisition means in real time, means for converting the received video information into an analyzable information format, analysis means for detecting abnormal behavior based on the converted video information, and means for notifying a mobile information terminal when abnormal behavior is detected. This enables the user to immediately detect anomalies through the monitoring system, confirm the situation via a mobile information terminal, and respond quickly.

[0271] "Information acquisition means" is a general term for devices and processes that acquire video information from the external environment.

[0272] "Means of receiving in real time" refers to technologies or methods for continuously incorporating video information into a system without delay.

[0273] An "analyzable information format" is a structured data format that a system needs to understand and process.

[0274] "Analysis means" refers to a process that includes algorithms and processing methods for analyzing converted video information and identifying abnormal behavior.

[0275] "Abnormal behavior" refers to actions that deviate from normal or expected behavioral patterns and carry identified risks within a system.

[0276] "Means of notifying relevant agencies" refers to methods or techniques for notifying pre-designated agencies of the situation when an anomaly is detected.

[0277] "Means of notifying of anomalies" refers to the function of a system that provides information to relevant parties when unusual behavior is detected.

[0278] A "portable information terminal" is a portable information processing device used by users to check system status and respond remotely.

[0279] To realize this invention, three parties—the server, the terminal, and the user—must each fulfill their respective roles.

[0280] The server receives video information in real time from information acquisition devices. Specifically, it collects data directly from surveillance cameras and sensors, converts it into a processable format, and transmits it to the terminal. Video streaming technology and data transfer protocols are used for this process.

[0281] The terminal analyzes the video information received from the server and detects abnormal behavior. When analyzing, an algorithm that detects deviations from the normal behavior pattern based on learning data is used. For this analysis, an image processing library such as OpenCV is used to identify situations such as illegal intrusion, fire occurrence, and fall accidents.

[0282] When abnormal behavior is detected, the terminal immediately reports it to the relevant institution and sends a notification to the user via the mobile information terminal. This notification includes an image of the scene and a situation description, enabling the user to immediately check the situation and take action.

[0283] The user uses the mobile information terminal to check the status of the monitoring system from a remote location. When an abnormality is detected, an alert is immediately sent, allowing the user to quickly grasp the situation at the scene and take necessary responses and notifications. As an example of a prompt sentence using generative AI, there is a sentence like "Please generate a scenario where a surveillance camera installed in an office detects an illegal intrusion at night. Also include the content of the system's notification and the flow of the administrator's response."

[0284] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0285] Step 1:

[0286] The server receives video information in real time from the information acquisition means. The input is video data from a surveillance camera, and initial processing is performed to transmit this in a predetermined data format to the terminal. The output is video data converted into an analyzable format. The server stably receives data through the network and performs compression and encoding to enable reliable data transfer.

[0287] Step 2:

[0288] The terminal prepares the video data received from the server into an analyzable format. The input is the video data sent from the server, which is decompressed and formatted as needed. The output is a dataset in a format usable by the analysis engine. Here, for example, consecutive frames are extracted from a JPEG image and prepared for analysis.

[0289] Step 3:

[0290] The device analyzes abnormal behavior using a prepared dataset. The input is transformed sequential frame data, and the output is the identification and location information of the detected abnormal behavior. Data processing is performed by running a motion detection algorithm using OpenCV. The device evaluates the differences in movement between frames and the length of time spent stationary, and detects deviations from the set normal behavior pattern.

[0291] Step 4:

[0292] When an anomaly is detected, the device initiates the process of notifying the relevant authorities. The input is the result of the detected abnormal behavior, and the output is the report content and a video clip. The device automatically compiles the necessary information and transmits it via email or API. Specifically, it generates a notification that includes a portion of the video, the time of occurrence, and location information.

[0293] Step 5:

[0294] The terminal notifies the mobile device of detected anomalies. Inputs are the detection results and video clips of the abnormal behavior, while output is an alert sent to the user. The terminal uses a communication protocol to send push notifications to the mobile device, allowing the user to immediately check the situation. As a concrete example, a notification appears on the user's smartphone, and they can view the on-site video through the application.

[0295] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0296] This invention relates to a system that uses video and audio data obtained from information acquisition devices such as surveillance cameras and microphones to monitor the safety of the elderly and infants in real time. This system incorporates an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection.

[0297] This system begins with a server collecting video and audio data received from an information acquisition device. The server then transmits this data to a terminal, where it is formatted into an analyzable format. The generating AI on the terminal uses the video data to detect abnormal behavior and simultaneously uses an emotion engine to analyze the user's emotions from the audio data.

[0298] The emotion engine uses emotion recognition technology to identify a user's emotional state from their tone of voice and speaking style. For example, it evaluates emotions such as whether an elderly person is feeling anxious or whether an infant is showing signs of sudden crying, and comprehensively detects signs of abnormality. If an abnormality is detected, the device uses that data to make an emergency call to the relevant authorities. If necessary, parts of the relevant video or audio will be attached to the call.

[0299] Users are immediately notified of the results of anomaly detection and emotion recognition. Through the mobile app, users can monitor the situation on-site and the evolution of emotions in real time. This feature enables faster and more appropriate responses, allowing for additional instructions and adjustments as needed.

[0300] As a specific example, assume a case where a solitary elderly person suddenly expresses uneasiness. At this time, the terminal analyzes video and audio data and determines that it is uneasiness. Based on this, an alert is sent to appropriate relevant parties, and if necessary, medical staff can be dispatched, etc. With this system, advanced anomaly detection incorporating emotional changes becomes possible, providing a support system for living daily life with more peace of mind.

[0301] The following explains the processing flow.

[0302] Step 1:

[0303] The server receives video and audio data in real time from the information acquisition device. Thereby, visual and auditory information of the monitoring target is acquired.

[0304] Step 2:

[0305] The server sends the received data to the terminal. The terminal converts this data into a format that can be analyzed and prepares for the next processing.

[0306] Step 3:

[0307] Based on the converted video data, the terminal uses a generation AI to analyze abnormal behavior. The AI compares with past behavior patterns and detects behaviors and movements different from normal.

[0308] Step 4:

[0309] Based on the audio data, the terminal analyzes the user's emotions using an emotion engine. The pitch and tone of the voice are analyzed to evaluate emotions such as uneasiness and stress.

[0310] Step 5:

[0311] The terminal integrates the results of video and emotion analysis and comprehensively determines the presence or absence of abnormalities. If an abnormality is detected, it triggers the next action.

[0312] Step 6:

[0313] The terminal uses anomaly detection data to send an emergency alert to nearby relevant organizations. In addition to conventional messages, the alert can also include analyzed emotional information.

[0314] Step 7:

[0315] The device instantly sends notifications of anomaly detection and emotion recognition to the user via a mobile app. This allows the user to immediately check the situation and take action.

[0316] Step 8:

[0317] The user receives a notification, opens the app, and checks the video and audio from the scene. This allows the user to understand the situation and send out necessary instructions.

[0318] (Example 2)

[0319] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0320] In modern times, effectively monitoring the safety of the elderly and infants is a critical challenge. Traditional monitoring systems have limitations in detecting abnormal behavior and require further improvement. Furthermore, they cannot capture emotional changes, potentially missing signs of abnormality. Overcoming these challenges and providing a safer living environment is essential.

[0321] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0322] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, means for detecting abnormal behavior using a generated AI model based on the converted data, means for analyzing emotional states from audio data based on emotion recognition technology, means for notifying relevant organizations when abnormal behavior or anxiety is detected, and means for notifying relevant parties of the abnormality and allowing them to check the situation in real time on a mobile terminal. This enables comprehensive safety management that takes into account not only the detection of abnormal behavior but also changes in emotions.

[0323] An "information acquisition device" is a device that collects data such as video and audio and provides it to a server.

[0324] "Real-time" refers to a state where information is processed and analyzed as soon as it is generated, and the results are reflected immediately.

[0325] An "analyzable format" refers to a format in which collected data has been appropriately processed and can be analyzed using AI models or other tools.

[0326] A "generative AI model" is an artificial intelligence model that recognizes patterns from new data based on previously learned data and performs specific tasks.

[0327] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns and is an important indicator that should be monitored.

[0328] "Emotion recognition technology" refers to technology that identifies a person's emotional state based on voice and text data.

[0329] "Reporting" refers to the act of informing the relevant authorities about information regarding detected anomalies.

[0330] A "mobile device" is a device that a user can carry with them and is used to receive information and perform operations via applications.

[0331] "Related organizations" refers to public or private organizations that provide support or response when an anomaly occurs.

[0332] Embodiments of this invention include the following systems and functions.

[0333] The system is initially composed of a server that receives video and audio data in real time using information acquisition devices such as surveillance cameras and microphones. The server receives the data via a communication interface and immediately transmits it to terminals. The terminals use a data conversion module to process the received data into a format that can be analyzed.

[0334] The device is equipped with a generative AI model for analyzing video data, enabling the detection of abnormal behavior. Furthermore, emotion recognition technology is used for analyzing audio data. This technology analyzes the tone and speed of speech to identify the user's emotional state. This enables advanced safety management that captures not only abnormal movements but also emotional changes.

[0335] If an anomaly is detected, the device automatically notifies the relevant authorities. The notification includes data on the detected abnormal behavior and emotional changes. Furthermore, the user is immediately notified of the anomaly, and this information can be viewed on their mobile device. In this way, the user can understand the situation in real time and take necessary actions quickly.

[0336] As a concrete example, in a living environment with an elderly person living alone, this system could detect an increase in the elderly person's anxiety and promptly notify care services or medical institutions. This would allow the user to live their daily life with greater peace of mind. An example of a prompt message would be, "Detect anxiety from the elderly person's tone of voice and suggest appropriate measures."

[0337] Ultimately, this system aims to leverage its advanced data processing capabilities to support users' lives while ensuring safety.

[0338] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0339] Step 1:

[0340] The server receives video and audio data in real time from surveillance cameras and microphones. The input data consists of stream data from the cameras and microphones, which is acquired by a network module within the server. The output is the collected raw video and audio data. Specifically, the server uses a communication protocol to ensure uninterrupted data reception.

[0341] Step 2:

[0342] The server sends the received data to the terminal. The input is the raw data mentioned earlier, and a communication format conversion is performed to convert it into a format suitable for transmission. The output is data packets grouped into an appropriate batch size. In actual operation, the server uses its data transmission function to ensure stable data transfer.

[0343] Step 3:

[0344] The terminal formats the received data into a format that can be analyzed. The input is data packets sent from the server, which are processed by the data conversion module. The output is data formatted for analysis, such as video frames and audio clips. Specifically, the terminal performs frame extraction and noise reduction before analyzing the data.

[0345] Step 4:

[0346] The on-device AI detects abnormal behavior. The input is formatted video data, and the AI ​​model is executed. The output is data of the detected abnormal behavior. Specifically, the AI ​​model analyzes the movement pattern for each frame and captures movements that deviate from normal movement patterns.

[0347] Step 5:

[0348] The device uses an emotion engine to analyze emotional states using voice data. The input is formatted voice data to which emotion recognition technology is applied. The output is information on the analyzed emotional state. Specifically, the model extracts embedded features such as voice tone and speech rate to estimate the emotional state.

[0349] Step 6:

[0350] The device will notify the relevant authorities if it detects abnormal behavior or anxiety. The input is the data on abnormalities and emotions obtained in the previous step, and a notification message is generated based on this data. The output is an emergency alert sent to the designated contact. Specifically, the message generation program attaches the relevant data to ensure rapid notification.

[0351] Step 7:

[0352] Users receive anomaly notifications and check the real-time status on their mobile devices. The input is notification data sent from the device, which allows for understanding the current situation. The output is a notification display and an interactive interface that the user can view and operate. Specifically, the mobile app displays notifications as pop-ups and provides functions to visualize video and sentiment analysis results.

[0353] (Application Example 2)

[0354] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0355] To ensure the safety of the elderly and infants in real time, it is necessary not only to detect abnormal behavior but also to quickly capture changes in their emotions. However, conventional technology is limited to detecting abnormal behavior and has the drawback of not being able to detect early signs of abnormality through emotion analysis from voice.

[0356] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0357] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, and means for detecting abnormal behavior from the converted data and analyzing emotional states from the audio data. This enables rapid and appropriate responses through highly accurate anomaly detection and emotion analysis.

[0358] An "information acquisition device" is a device that has the function of collecting video data and audio data.

[0359] "Video data" refers to image information acquired by surveillance cameras and other video devices.

[0360] "Audio data" refers to sound information acquired by a microphone or similar device.

[0361] A "means of receiving data in real time" refers to a mechanism that processes acquired data almost simultaneously.

[0362] "Means of converting to an analyzable format" refers to a function that processes received data into a format that can be processed by an analysis engine.

[0363] An "analysis means for detecting abnormal behavior" is a mechanism that analyzes video data to identify deviations from normal behavior.

[0364] A "means for analyzing emotional states" refers to a mechanism that analyzes voice data to identify the user's emotions.

[0365] "A means of notifying mobile devices and reporting to relevant organizations" refers to a system that transmits information to portable communication devices or relevant departments when an anomaly is detected.

[0366] "Means for transmitting video and audio of abnormal behavior to relevant organizations" refers to a function that transmits data that serves as evidence of detected abnormalities to public or designated organizations.

[0367] The system necessary to implement this invention consists of an information acquisition device, a server, a terminal, and a mobile device used by the user. The server receives video and audio data acquired from surveillance cameras and microphones in real time. The received data is converted into an analyzable format, and analysis is performed on the server to detect abnormal behavior. Video analysis libraries such as OpenCV are used for this analysis.

[0368] Furthermore, voice analysis technology is applied to the analysis of voice data to identify the user's emotional state from their tone of voice and speaking style. For example, by using voice recognition services such as Azure Cognitive Services, changes in emotion can be analyzed in real time. This combination of emotion recognition and abnormal behavior detection enables highly accurate safety monitoring.

[0369] The user's device is immediately notified of any detected anomalies. Push notifications are sent via Firebase Cloud Messaging, allowing the user to take appropriate action based on the notification. If necessary, the relevant authorities will also be notified. In such cases, video and audio recordings capturing abnormal behavior or emotional changes may be attached.

[0370] For example, if an elderly person begins to feel anxious at home, the system analyzes their emotions from their voice and detects it as an abnormality. This immediately sends a notification to the family member, allowing for a quick response while viewing the video and audio. This system provides a safe and secure monitoring environment.

[0371] An example of a prompt for using a generative AI model to more precisely analyze abnormal behavior would be: "Describe a system that monitors the daily activities of elderly people, detects emotional changes, and provides real-time notifications." This allows the AI ​​model to perform a detailed analysis and enable flexible responses.

[0372] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0373] Step 1:

[0374] The server receives video and audio data in real time from surveillance cameras and microphones. The input is raw data from the cameras and microphones, while the output is data temporarily stored within the server. This data is processed as a stream to maintain real-time performance.

[0375] Step 2:

[0376] The server converts the received raw data into a format that can be analyzed. The input is raw video and audio data, and the output is data in a format that the analysis engine can process. The OpenCV library is used to convert the video data to a standard image format, and the audio is converted to WAV or MP3 format.

[0377] Step 3:

[0378] The server detects abnormal behavior based on the converted video data. The input is the converted video data, and the output is the evaluation result regarding the abnormal behavior. OpenCV is used to analyze the movement and compare it with predetermined abnormal behavior patterns.

[0379] Step 4:

[0380] The server analyzes audio data to identify emotional states. The input is the converted audio data, and the output is the detected emotion label. Speech recognition technologies such as Azure Cognitive Services are used to analyze emotions from voice tone and speed.

[0381] Step 5:

[0382] The server determines whether an anomaly has occurred based on the detection results and outputs the result. The input is the evaluation result of abnormal behavior and the result of sentiment analysis, and the output is a flag indicating the presence or absence of an anomaly and its detailed information. If an anomaly is detected, the server prepares to send a notification request to the mobile device.

[0383] Step 6:

[0384] The server pushes information about the anomaly to the user's mobile device. The input is detailed information about the anomaly, and the output is a notification message to the user. Firebase Cloud Messaging is used for rapid notification.

[0385] Step 7:

[0386] Users receive notifications and view detailed video and audio via their mobile devices. The input is the notification message sent from the server, and the output is the video and audio content of the anomaly displayed on the device. This allows for rapid countermeasures to be taken.

[0387] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0388] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0389] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0390] [Third Embodiment]

[0391] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0392] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0393] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0394] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0395] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0397] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0398] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0399] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0400] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0401] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0402] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0403] This invention relates to a system that detects abnormal behavior by receiving video data in real time from information acquisition devices such as surveillance cameras and analyzing that video data. The system consists of three components: a server, a terminal, and a user, each of which is responsible for a specific function.

[0404] The server continuously receives video data from the information acquisition device. The received data is sent to a terminal, where it is converted into an analyzable format. The terminal uses a specific algorithm to analyze the video data and detect abnormal behavior. For example, falls and prolonged periods of remaining stationary in the same position are among the behaviors that can be detected.

[0405] If abnormal behavior is detected, the device immediately initiates a notification process to the relevant authorities. This notification includes the location and timing of the abnormal behavior, and, if necessary, a portion of the video footage. Notifying relevant parties allows for quick sharing of the situation and enables a swift response.

[0406] Users can remotely monitor the status of the monitoring system using a mobile application. When an anomaly is detected, an alert is sent, allowing for immediate on-site verification and necessary action. For example, even if a user is in a remote location, they can view on-site video through the app and receive support to respond appropriately.

[0407] As a concrete example, consider a case where an elderly person living alone falls in their living room. When a surveillance camera detects the anomaly, the data is analyzed on a terminal via a server, confirming the fall. The terminal immediately notifies emergency services and sends an alert to registered family members, prompting a quick response. This system makes it possible to more reliably monitor the safety of the elderly and infants and to respond quickly in emergencies.

[0408] The following describes the processing flow.

[0409] Step 1:

[0410] The server receives video data in real time from the information acquisition device. This provides continuous visual information within the monitoring area.

[0411] Step 2:

[0412] The server converts the received video data into an analyzable format. Pre-processing, such as image resolution adjustment and noise reduction, is performed during this stage.

[0413] Step 3:

[0414] The device acquires the converted video data and uses a generating AI to analyze abnormal behavior. The AI ​​compares it to learned normal behavior patterns to determine if there are any deviations.

[0415] Step 4:

[0416] If the terminal determines, based on the analysis results, that abnormal behavior has occurred, it will save the data and activate the means to notify the appropriate authorities. Specifically, an emergency notification message will be created and sent.

[0417] Step 5:

[0418] The device initiates the notification process to relevant parties. Users receive an alert via the mobile app, which includes a detailed notification of the situation.

[0419] Step 6:

[0420] Based on alerts received by the user, on-site video footage can be viewed through the app. This allows the user to respond quickly and send additional instructions to relevant parties as needed.

[0421] (Example 1)

[0422] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0423] For the elderly and individuals requiring specific risk management, there is a need for more efficient and rapid response monitoring systems that provide safety and security. However, current technology lacks the ability to detect abnormal behavior in real time and immediately notify the appropriate organizations and individuals. Furthermore, analyzing the data requires multiple processing steps, and the lack of system integration to smoothly carry out these processes is a challenge.

[0424] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0425] In this invention, the server is a device that receives video data acquired from an information acquisition device in real time and transmits it to a data processing device; a device that converts the received video data into an analyzable data format to enable analysis in the next stage; and an analysis device that detects abnormal behavior from the converted data using a specific algorithm. This makes it possible to quickly detect abnormal behavior and immediately notify relevant organizations and individuals.

[0426] An "information acquisition device" is a hardware device used to acquire video data from a monitored object.

[0427] A "data processing device" is a device that converts received data into an analyzable format and then performs further analysis on it.

[0428] An "analysis device" is a device used to detect abnormal behavior from converted video data, and it performs analysis using a specific algorithm.

[0429] A "communication device" is a device used to notify relevant organizations of information when abnormal behavior is detected.

[0430] A "notification device" is a device that notifies registered parties of information when an anomaly is detected.

[0431] A "specific algorithm" is a computational method used to detect abnormal behavior from data.

[0432] "Data format" refers to the data structure and encoding necessary for analyzing video data.

[0433] This invention is a safety monitoring system for the elderly and individuals requiring specific risk management. The system consists of three components: a server, a terminal, and a user.

[0434] The server receives video data in real time from information acquisition devices such as surveillance cameras. This data is first transmitted to a data processing unit. The server is equipped with high-performance communication equipment to ensure stable data reception.

[0435] The terminal converts the video data sent from the server into an analyzable data format. Here, it compresses the data using a specific codec and adjusts the resolution as needed. In particular, it runs a specific algorithm to analyze abnormal behavior and detect it rapidly. The algorithm used by the analysis device extracts abnormal patterns based on stored training data and recognizes anomalies in real time.

[0436] When abnormal behavior is detected, the device notifies the relevant authorities via a communication device. At the same time, it also notifies registered contacts to promptly take necessary action. This enables a smooth response in emergencies, such as when an elderly person falls.

[0437] Users can remotely monitor the system's operational status using a mobile application. The application immediately displays an alert when an anomaly is detected, allowing users to understand the situation on-site. Users can also request additional information from the app as needed.

[0438] An example of a prompt is, "Please describe the fall detection system for the elderly. This system analyzes video data in real time and detects abnormal behavior." Using this prompt allows for efficient communication of the system's purpose and usage conditions to the generated AI model.

[0439] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0440] Step 1:

[0441] The server receives video data in real time from the information acquisition device. The input is raw data from surveillance cameras, and the output generates stream data for transmission to the data processing device. Specifically, it integrates data from multiple cameras and utilizes a communication protocol to achieve consistent data distribution.

[0442] Step 2:

[0443] The terminal receives raw stream data sent from the server and converts it into a parseable format. The input is stream data from the server, and the output is data in a format suitable for the analysis algorithm. Specifically, it performs data compression using a codec and adjusts the resolution as needed, then sends the converted data to the next processing step.

[0444] Step 3:

[0445] The device analyzes the converted data using a specific algorithm to detect abnormal behavior. The input is data in a parsable format, and the output is the result of detecting abnormal behavior. The specific operation involves using a machine learning model to recognize anomalies in real time based on training data. Based on this analysis, the type and severity of the anomaly are identified.

[0446] Step 4:

[0447] The terminal immediately notifies the relevant authorities using its communication device upon detecting abnormal behavior. The input is the result of the detected abnormal behavior, and the output is the notification message to the relevant authorities. Specific actions include transmitting information such as the location and time the abnormality occurred, and, if necessary, attaching a portion of the video footage.

[0448] Step 5:

[0449] Users receive anomaly notifications via a mobile application and check the situation on-site. Input is anomaly notifications from the device, and output is an alert displayed to the user. Specifically, users can use the app to view detailed video footage and issue instructions or additional notifications to the site as needed.

[0450] (Application Example 1)

[0451] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0452] In recent years, with the increasing importance of security, there has been a growing demand for monitoring systems to detect abnormal behavior. However, many conventional systems lack sufficient remote means to immediately confirm the situation on-site and respond, resulting in a lack of rapid and appropriate response. In particular, the difficulty for users to quickly grasp the situation and respond via mobile devices can lead to security vulnerabilities.

[0453] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0454] In this invention, the server includes means for receiving video information acquired from an information acquisition means in real time, means for converting the received video information into an analyzable information format, analysis means for detecting abnormal behavior based on the converted video information, and means for notifying a mobile information terminal when abnormal behavior is detected. This enables the user to immediately detect anomalies through the monitoring system, confirm the situation via a mobile information terminal, and respond quickly.

[0455] "Information acquisition means" is a general term for devices and processes that acquire video information from the external environment.

[0456] "Means of receiving in real time" refers to technologies or methods for continuously incorporating video information into a system without delay.

[0457] An "analyzable information format" is a structured data format that a system needs to understand and process.

[0458] "Analysis means" refers to a process that includes algorithms and processing methods for analyzing converted video information and identifying abnormal behavior.

[0459] "Abnormal behavior" refers to actions that deviate from normal or expected behavioral patterns and carry identified risks within a system.

[0460] "Means of notifying relevant agencies" refers to methods or techniques for notifying pre-designated agencies of the situation when an anomaly is detected.

[0461] "Means of notifying of anomalies" refers to the function of a system that provides information to relevant parties when unusual behavior is detected.

[0462] A "portable information terminal" is a portable information processing device used by users to check system status and respond remotely.

[0463] To realize this invention, three parties—the server, the terminal, and the user—must each fulfill their respective roles.

[0464] The server receives video information in real time from information acquisition devices. Specifically, it collects data directly from surveillance cameras and sensors, converts it into a processable format, and transmits it to the terminal. Video streaming technology and data transfer protocols are used for this process.

[0465] The terminal analyzes video information received from the server to detect abnormal behavior. The analysis uses an algorithm that detects deviations from normal behavior patterns based on training data. Image processing libraries such as OpenCV are used in this analysis to identify situations such as illegal entry, fires, and falls.

[0466] When abnormal behavior is detected, the device immediately notifies the relevant authorities and sends a notification to the user via the mobile device. This notification includes images of the scene and a description of the situation, allowing the user to immediately check the situation and take appropriate action.

[0467] Users can remotely check the status of the monitoring system using a mobile device. When an anomaly is detected, an alert is immediately sent, allowing users to quickly understand the situation on-site and take necessary responses or notifications. An example of a prompt message generated using AI is: "Generate a scenario in which a surveillance camera installed in an office detects an illegal intrusion at night. Include the system notification content and the administrator's response flow."

[0468] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0469] Step 1:

[0470] The server receives video information in real time from the information acquisition means. The input is video data from surveillance cameras, and the server performs initial processing to send it to the terminal in a predetermined data format. The output is video data converted into an analyzable format. The server receives data stably over the network and performs compression and encoding to enable reliable data transfer.

[0471] Step 2:

[0472] The terminal prepares the video data received from the server into an analyzable format. The input is the video data sent from the server, which is decompressed and formatted as needed. The output is a dataset in a format usable by the analysis engine. Here, for example, consecutive frames are extracted from a JPEG image and prepared for analysis.

[0473] Step 3:

[0474] The device analyzes abnormal behavior using a prepared dataset. The input is transformed sequential frame data, and the output is the identification and location information of the detected abnormal behavior. Data processing is performed by running a motion detection algorithm using OpenCV. The device evaluates the differences in movement between frames and the length of time spent stationary, and detects deviations from the set normal behavior pattern.

[0475] Step 4:

[0476] When an anomaly is detected, the device initiates the process of notifying the relevant authorities. The input is the result of the detected abnormal behavior, and the output is the report content and a video clip. The device automatically compiles the necessary information and transmits it via email or API. Specifically, it generates a notification that includes a portion of the video, the time of occurrence, and location information.

[0477] Step 5:

[0478] The terminal notifies the mobile device of detected anomalies. Inputs are the detection results and video clips of the abnormal behavior, while output is an alert sent to the user. The terminal uses a communication protocol to send push notifications to the mobile device, allowing the user to immediately check the situation. As a concrete example, a notification appears on the user's smartphone, and they can view the on-site video through the application.

[0479] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0480] This invention relates to a system that uses video and audio data obtained from information acquisition devices such as surveillance cameras and microphones to monitor the safety of the elderly and infants in real time. This system incorporates an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection.

[0481] This system begins with a server collecting video and audio data received from an information acquisition device. The server then transmits this data to a terminal, where it is formatted into an analyzable format. The generating AI on the terminal uses the video data to detect abnormal behavior and simultaneously uses an emotion engine to analyze the user's emotions from the audio data.

[0482] The emotion engine uses emotion recognition technology to identify a user's emotional state from their tone of voice and speaking style. For example, it evaluates emotions such as whether an elderly person is feeling anxious or whether an infant is showing signs of sudden crying, and comprehensively detects signs of abnormality. If an abnormality is detected, the device uses that data to make an emergency call to the relevant authorities. If necessary, parts of the relevant video or audio will be attached to the call.

[0483] Users are immediately notified of the results of anomaly detection and emotion recognition. Through the mobile app, users can monitor the situation on-site and the evolution of emotions in real time. This feature enables faster and more appropriate responses, allowing for additional instructions and adjustments as needed.

[0484] As a concrete example, consider a scenario where an elderly person living alone suddenly expresses anxiety. In this case, the terminal analyzes video and audio data and determines that the person is experiencing anxiety. Based on this, it can send an alert to the appropriate contacts and, if necessary, dispatch medical staff. This system enables advanced anomaly detection that incorporates changes in emotions, providing a support system that allows people to live their daily lives with greater peace of mind.

[0485] The following describes the processing flow.

[0486] Step 1:

[0487] The server receives video and audio data in real time from the information acquisition device. This allows for the acquisition of visual and auditory information about the monitored target.

[0488] Step 2:

[0489] The server sends the received data to the terminal. The terminal converts this data into a parseable format and prepares for the next processing step.

[0490] Step 3:

[0491] The terminal uses a generating AI to analyze abnormal behavior based on the converted video data. The AI ​​compares the current behavior with past behavioral patterns to detect unusual actions and movements.

[0492] Step 4:

[0493] The device uses an emotion engine to analyze the user's emotions based on voice data. It analyzes the pitch and tone of the voice to evaluate emotions such as anxiety and stress.

[0494] Step 5:

[0495] The device integrates video and emotion analysis results to make a comprehensive judgment about the presence or absence of anomalies. If an anomaly is detected, it triggers the next action.

[0496] Step 6:

[0497] The terminal uses anomaly detection data to send an emergency alert to nearby relevant organizations. In addition to conventional messages, the alert can also include analyzed emotional information.

[0498] Step 7:

[0499] The device instantly sends notifications of anomaly detection and emotion recognition to the user via a mobile app. This allows the user to immediately check the situation and take action.

[0500] Step 8:

[0501] The user receives a notification, opens the app, and checks the video and audio from the scene. This allows the user to understand the situation and send out necessary instructions.

[0502] (Example 2)

[0503] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0504] In modern times, effectively monitoring the safety of the elderly and infants is a critical challenge. Traditional monitoring systems have limitations in detecting abnormal behavior and require further improvement. Furthermore, they cannot capture emotional changes, potentially missing signs of abnormality. Overcoming these challenges and providing a safer living environment is essential.

[0505] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0506] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, means for detecting abnormal behavior using a generated AI model based on the converted data, means for analyzing emotional states from audio data based on emotion recognition technology, means for notifying relevant organizations when abnormal behavior or anxiety is detected, and means for notifying relevant parties of the abnormality and allowing them to check the situation in real time on a mobile terminal. This enables comprehensive safety management that takes into account not only the detection of abnormal behavior but also changes in emotions.

[0507] An "information acquisition device" is a device that collects data such as video and audio and provides it to a server.

[0508] "Real-time" refers to a state where information is processed and analyzed as soon as it is generated, and the results are reflected immediately.

[0509] An "analyzable format" refers to a format in which collected data has been appropriately processed and can be analyzed using AI models or other tools.

[0510] A "generative AI model" is an artificial intelligence model that recognizes patterns from new data based on previously learned data and performs specific tasks.

[0511] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns and is an important indicator that should be monitored.

[0512] "Emotion recognition technology" refers to technology that identifies a person's emotional state based on voice and text data.

[0513] "Reporting" refers to the act of informing the relevant authorities about information regarding detected anomalies.

[0514] A "mobile device" is a device that a user can carry with them and is used to receive information and perform operations via applications.

[0515] "Related organizations" refers to public or private organizations that provide support or response when an anomaly occurs.

[0516] Embodiments of this invention include the following systems and functions.

[0517] The system is initially composed of a server that receives video and audio data in real time using information acquisition devices such as surveillance cameras and microphones. The server receives the data via a communication interface and immediately transmits it to terminals. The terminals use a data conversion module to process the received data into a format that can be analyzed.

[0518] The device is equipped with a generative AI model for analyzing video data, enabling the detection of abnormal behavior. Furthermore, emotion recognition technology is used for analyzing audio data. This technology analyzes the tone and speed of speech to identify the user's emotional state. This enables advanced safety management that captures not only abnormal movements but also emotional changes.

[0519] If an anomaly is detected, the device automatically notifies the relevant authorities. The notification includes data on the detected abnormal behavior and emotional changes. Furthermore, the user is immediately notified of the anomaly, and this information can be viewed on their mobile device. In this way, the user can understand the situation in real time and take necessary actions quickly.

[0520] As a concrete example, in a living environment with an elderly person living alone, this system could detect an increase in the elderly person's anxiety and promptly notify care services or medical institutions. This would allow the user to live their daily life with greater peace of mind. An example of a prompt message would be, "Detect anxiety from the elderly person's tone of voice and suggest appropriate measures."

[0521] Ultimately, this system aims to leverage its advanced data processing capabilities to support users' lives while ensuring safety.

[0522] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0523] Step 1:

[0524] The server receives video and audio data in real time from surveillance cameras and microphones. The input data consists of stream data from the cameras and microphones, which is acquired by a network module within the server. The output is the collected raw video and audio data. Specifically, the server uses a communication protocol to ensure uninterrupted data reception.

[0525] Step 2:

[0526] The server sends the received data to the terminal. The input is the raw data mentioned earlier, and a communication format conversion is performed to convert it into a format suitable for transmission. The output is data packets grouped into an appropriate batch size. In actual operation, the server uses its data transmission function to ensure stable data transfer.

[0527] Step 3:

[0528] The terminal formats the received data into a format that can be analyzed. The input is data packets sent from the server, which are processed by the data conversion module. The output is data formatted for analysis, such as video frames and audio clips. Specifically, the terminal performs frame extraction and noise reduction before analyzing the data.

[0529] Step 4:

[0530] The on-device AI detects abnormal behavior. The input is formatted video data, and the AI ​​model is executed. The output is data of the detected abnormal behavior. Specifically, the AI ​​model analyzes the movement pattern for each frame and captures movements that deviate from normal movement patterns.

[0531] Step 5:

[0532] The device uses an emotion engine to analyze emotional states using voice data. The input is formatted voice data to which emotion recognition technology is applied. The output is information on the analyzed emotional state. Specifically, the model extracts embedded features such as voice tone and speech rate to estimate the emotional state.

[0533] Step 6:

[0534] The device will notify the relevant authorities if it detects abnormal behavior or anxiety. The input is the data on abnormalities and emotions obtained in the previous step, and a notification message is generated based on this data. The output is an emergency alert sent to the designated contact. Specifically, the message generation program attaches the relevant data to ensure rapid notification.

[0535] Step 7:

[0536] Users receive anomaly notifications and check the real-time status on their mobile devices. The input is notification data sent from the device, which allows for understanding the current situation. The output is a notification display and an interactive interface that the user can view and operate. Specifically, the mobile app displays notifications as pop-ups and provides functions to visualize video and sentiment analysis results.

[0537] (Application Example 2)

[0538] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0539] To ensure the safety of the elderly and infants in real time, it is necessary not only to detect abnormal behavior but also to quickly capture changes in their emotions. However, conventional technology is limited to detecting abnormal behavior and has the drawback of not being able to detect early signs of abnormality through emotion analysis from voice.

[0540] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0541] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, and means for detecting abnormal behavior from the converted data and analyzing emotional states from the audio data. This enables rapid and appropriate responses through highly accurate anomaly detection and emotion analysis.

[0542] An "information acquisition device" is a device that has the function of collecting video data and audio data.

[0543] "Video data" refers to image information acquired by surveillance cameras and other video devices.

[0544] "Audio data" refers to sound information acquired by a microphone or similar device.

[0545] A "means of receiving data in real time" refers to a mechanism that processes acquired data almost simultaneously.

[0546] "Means of converting to an analyzable format" refers to a function that processes received data into a format that can be processed by an analysis engine.

[0547] An "analysis means for detecting abnormal behavior" is a mechanism that analyzes video data to identify deviations from normal behavior.

[0548] A "means for analyzing emotional states" refers to a mechanism that analyzes voice data to identify the user's emotions.

[0549] "A means of notifying mobile devices and reporting to relevant organizations" refers to a system that transmits information to portable communication devices or relevant departments when an anomaly is detected.

[0550] "Means for transmitting video and audio of abnormal behavior to relevant organizations" refers to a function that transmits data that serves as evidence of detected abnormalities to public or designated organizations.

[0551] The system necessary to implement this invention consists of an information acquisition device, a server, a terminal, and a mobile device used by the user. The server receives video and audio data acquired from surveillance cameras and microphones in real time. The received data is converted into an analyzable format, and analysis is performed on the server to detect abnormal behavior. Video analysis libraries such as OpenCV are used for this analysis.

[0552] Furthermore, voice analysis technology is applied to the analysis of voice data to identify the user's emotional state from their tone of voice and speaking style. For example, by using voice recognition services such as Azure Cognitive Services, changes in emotion can be analyzed in real time. This combination of emotion recognition and abnormal behavior detection enables highly accurate safety monitoring.

[0553] The user's device is immediately notified of any detected anomalies. Push notifications are sent via Firebase Cloud Messaging, allowing the user to take appropriate action based on the notification. If necessary, the relevant authorities will also be notified. In such cases, video and audio recordings capturing abnormal behavior or emotional changes may be attached.

[0554] For example, if an elderly person begins to feel anxious at home, the system analyzes their emotions from their voice and detects it as an abnormality. This immediately sends a notification to the family member, allowing for a quick response while viewing the video and audio. This system provides a safe and secure monitoring environment.

[0555] An example of a prompt for using a generative AI model to more precisely analyze abnormal behavior would be: "Describe a system that monitors the daily activities of elderly people, detects emotional changes, and provides real-time notifications." This allows the AI ​​model to perform a detailed analysis and enable flexible responses.

[0556] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0557] Step 1:

[0558] The server receives video and audio data in real time from surveillance cameras and microphones. The input is raw data from the cameras and microphones, while the output is data temporarily stored within the server. This data is processed as a stream to maintain real-time performance.

[0559] Step 2:

[0560] The server converts the received raw data into a format that can be analyzed. The input is raw video and audio data, and the output is data in a format that the analysis engine can process. The OpenCV library is used to convert the video data to a standard image format, and the audio is converted to WAV or MP3 format.

[0561] Step 3:

[0562] The server detects abnormal behavior based on the converted video data. The input is the converted video data, and the output is the evaluation result regarding the abnormal behavior. OpenCV is used to analyze the movement and compare it with predetermined abnormal behavior patterns.

[0563] Step 4:

[0564] The server analyzes audio data to identify emotional states. The input is the converted audio data, and the output is the detected emotion label. Speech recognition technologies such as Azure Cognitive Services are used to analyze emotions from voice tone and speed.

[0565] Step 5:

[0566] The server determines whether an anomaly has occurred based on the detection results and outputs the result. The input is the evaluation result of abnormal behavior and the result of sentiment analysis, and the output is a flag indicating the presence or absence of an anomaly and its detailed information. If an anomaly is detected, the server prepares to send a notification request to the mobile device.

[0567] Step 6:

[0568] The server pushes information about the anomaly to the user's mobile device. The input is detailed information about the anomaly, and the output is a notification message to the user. Firebase Cloud Messaging is used for rapid notification.

[0569] Step 7:

[0570] Users receive notifications and view detailed video and audio via their mobile devices. The input is the notification message sent from the server, and the output is the video and audio content of the anomaly displayed on the device. This allows for rapid countermeasures to be taken.

[0571] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0572] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0573] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0574] [Fourth Embodiment]

[0575] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0576] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0577] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0578] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0579] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0580] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0581] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0582] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0583] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0584] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0585] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0586] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0587] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0588] This invention relates to a system that detects abnormal behavior by receiving video data in real time from information acquisition devices such as surveillance cameras and analyzing that video data. The system consists of three components: a server, a terminal, and a user, each of which is responsible for a specific function.

[0589] The server continuously receives video data from the information acquisition device. The received data is sent to a terminal, where it is converted into an analyzable format. The terminal uses a specific algorithm to analyze the video data and detect abnormal behavior. For example, falls and prolonged periods of remaining stationary in the same position are among the behaviors that can be detected.

[0590] If abnormal behavior is detected, the device immediately initiates a notification process to the relevant authorities. This notification includes the location and timing of the abnormal behavior, and, if necessary, a portion of the video footage. Notifying relevant parties allows for quick sharing of the situation and enables a swift response.

[0591] Users can remotely monitor the status of the monitoring system using a mobile application. When an anomaly is detected, an alert is sent, allowing for immediate on-site verification and necessary action. For example, even if a user is in a remote location, they can view on-site video through the app and receive support to respond appropriately.

[0592] As a concrete example, consider a case where an elderly person living alone falls in their living room. When a surveillance camera detects the anomaly, the data is analyzed on a terminal via a server, confirming the fall. The terminal immediately notifies emergency services and sends an alert to registered family members, prompting a quick response. This system makes it possible to more reliably monitor the safety of the elderly and infants and to respond quickly in emergencies.

[0593] The following describes the processing flow.

[0594] Step 1:

[0595] The server receives video data in real time from the information acquisition device. This provides continuous visual information within the monitoring area.

[0596] Step 2:

[0597] The server converts the received video data into an analyzable format. Pre-processing, such as image resolution adjustment and noise reduction, is performed during this stage.

[0598] Step 3:

[0599] The device acquires the converted video data and uses a generating AI to analyze abnormal behavior. The AI ​​compares it to learned normal behavior patterns to determine if there are any deviations.

[0600] Step 4:

[0601] If the terminal determines, based on the analysis results, that abnormal behavior has occurred, it will save the data and activate the means to notify the appropriate authorities. Specifically, an emergency notification message will be created and sent.

[0602] Step 5:

[0603] The device initiates the notification process to relevant parties. Users receive an alert via the mobile app, which includes a detailed notification of the situation.

[0604] Step 6:

[0605] Based on alerts received by the user, on-site video footage can be viewed through the app. This allows the user to respond quickly and send additional instructions to relevant parties as needed.

[0606] (Example 1)

[0607] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0608] For the elderly and individuals requiring specific risk management, there is a need for more efficient and rapid response monitoring systems that provide safety and security. However, current technology lacks the ability to detect abnormal behavior in real time and immediately notify the appropriate organizations and individuals. Furthermore, analyzing the data requires multiple processing steps, and the lack of system integration to smoothly carry out these processes is a challenge.

[0609] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0610] In this invention, the server is a device that receives video data acquired from an information acquisition device in real time and transmits it to a data processing device; a device that converts the received video data into an analyzable data format to enable analysis in the next stage; and an analysis device that detects abnormal behavior from the converted data using a specific algorithm. This makes it possible to quickly detect abnormal behavior and immediately notify relevant organizations and individuals.

[0611] An "information acquisition device" is a hardware device used to acquire video data from a monitored object.

[0612] A "data processing device" is a device that converts received data into an analyzable format and then performs further analysis on it.

[0613] An "analysis device" is a device used to detect abnormal behavior from converted video data, and it performs analysis using a specific algorithm.

[0614] A "communication device" is a device used to notify relevant organizations of information when abnormal behavior is detected.

[0615] A "notification device" is a device that notifies registered parties of information when an anomaly is detected.

[0616] A "specific algorithm" is a computational method used to detect abnormal behavior from data.

[0617] "Data format" refers to the data structure and encoding necessary for analyzing video data.

[0618] This invention is a safety monitoring system for the elderly and individuals requiring specific risk management. The system consists of three components: a server, a terminal, and a user.

[0619] The server receives video data in real time from information acquisition devices such as surveillance cameras. This data is first transmitted to a data processing unit. The server is equipped with high-performance communication equipment to ensure stable data reception.

[0620] The terminal converts the video data sent from the server into an analyzable data format. Here, it compresses the data using a specific codec and adjusts the resolution as needed. In particular, it runs a specific algorithm to analyze abnormal behavior and detect it rapidly. The algorithm used by the analysis device extracts abnormal patterns based on stored training data and recognizes anomalies in real time.

[0621] When abnormal behavior is detected, the device notifies the relevant authorities via a communication device. At the same time, it also notifies registered contacts to promptly take necessary action. This enables a smooth response in emergencies, such as when an elderly person falls.

[0622] Users can remotely monitor the system's operational status using a mobile application. The application immediately displays an alert when an anomaly is detected, allowing users to understand the situation on-site. Users can also request additional information from the app as needed.

[0623] An example of a prompt is, "Please describe the fall detection system for the elderly. This system analyzes video data in real time and detects abnormal behavior." Using this prompt allows for efficient communication of the system's purpose and usage conditions to the generated AI model.

[0624] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0625] Step 1:

[0626] The server receives video data in real time from the information acquisition device. The input is raw data from surveillance cameras, and the output generates stream data for transmission to the data processing device. Specifically, it integrates data from multiple cameras and utilizes a communication protocol to achieve consistent data distribution.

[0627] Step 2:

[0628] The terminal receives raw stream data sent from the server and converts it into a parseable format. The input is stream data from the server, and the output is data in a format suitable for the analysis algorithm. Specifically, it performs data compression using a codec and adjusts the resolution as needed, then sends the converted data to the next processing step.

[0629] Step 3:

[0630] The device analyzes the converted data using a specific algorithm to detect abnormal behavior. The input is data in a parsable format, and the output is the result of detecting abnormal behavior. The specific operation involves using a machine learning model to recognize anomalies in real time based on training data. Based on this analysis, the type and severity of the anomaly are identified.

[0631] Step 4:

[0632] The terminal immediately notifies the relevant authorities using its communication device upon detecting abnormal behavior. The input is the result of the detected abnormal behavior, and the output is the notification message to the relevant authorities. Specific actions include transmitting information such as the location and time the abnormality occurred, and, if necessary, attaching a portion of the video footage.

[0633] Step 5:

[0634] Users receive anomaly notifications via a mobile application and check the situation on-site. Input is anomaly notifications from the device, and output is an alert displayed to the user. Specifically, users can use the app to view detailed video footage and issue instructions or additional notifications to the site as needed.

[0635] (Application Example 1)

[0636] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0637] In recent years, with the increasing importance of security, there has been a growing demand for monitoring systems to detect abnormal behavior. However, many conventional systems lack sufficient remote means to immediately confirm the situation on-site and respond, resulting in a lack of rapid and appropriate response. In particular, the difficulty for users to quickly grasp the situation and respond via mobile devices can lead to security vulnerabilities.

[0638] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0639] In this invention, the server includes means for receiving video information acquired from an information acquisition means in real time, means for converting the received video information into an analyzable information format, analysis means for detecting abnormal behavior based on the converted video information, and means for notifying a mobile information terminal when abnormal behavior is detected. This enables the user to immediately detect anomalies through the monitoring system, confirm the situation via a mobile information terminal, and respond quickly.

[0640] "Information acquisition means" is a general term for devices and processes that acquire video information from the external environment.

[0641] "Means of receiving in real time" refers to technologies or methods for continuously incorporating video information into a system without delay.

[0642] An "analyzable information format" is a structured data format that a system needs to understand and process.

[0643] "Analysis means" refers to a process that includes algorithms and processing methods for analyzing converted video information and identifying abnormal behavior.

[0644] "Abnormal behavior" refers to actions that deviate from normal or expected behavioral patterns and carry identified risks within a system.

[0645] "Means of notifying relevant agencies" refers to methods or techniques for notifying pre-designated agencies of the situation when an anomaly is detected.

[0646] "Means of notifying of anomalies" refers to the function of a system that provides information to relevant parties when unusual behavior is detected.

[0647] A "portable information terminal" is a portable information processing device used by users to check system status and respond remotely.

[0648] To realize this invention, three parties—the server, the terminal, and the user—must each fulfill their respective roles.

[0649] The server receives video information in real time from information acquisition devices. Specifically, it collects data directly from surveillance cameras and sensors, converts it into a processable format, and transmits it to the terminal. Video streaming technology and data transfer protocols are used for this process.

[0650] The terminal analyzes video information received from the server to detect abnormal behavior. The analysis uses an algorithm that detects deviations from normal behavior patterns based on training data. Image processing libraries such as OpenCV are used in this analysis to identify situations such as illegal entry, fires, and falls.

[0651] When abnormal behavior is detected, the device immediately notifies the relevant authorities and sends a notification to the user via the mobile device. This notification includes images of the scene and a description of the situation, allowing the user to immediately check the situation and take appropriate action.

[0652] Users can remotely check the status of the monitoring system using a mobile device. When an anomaly is detected, an alert is immediately sent, allowing users to quickly understand the situation on-site and take necessary responses or notifications. An example of a prompt message generated using AI is: "Generate a scenario in which a surveillance camera installed in an office detects an illegal intrusion at night. Include the system notification content and the administrator's response flow."

[0653] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0654] Step 1:

[0655] The server receives video information in real time from the information acquisition means. The input is video data from surveillance cameras, and the server performs initial processing to send it to the terminal in a predetermined data format. The output is video data converted into an analyzable format. The server receives data stably over the network and performs compression and encoding to enable reliable data transfer.

[0656] Step 2:

[0657] The terminal prepares the video data received from the server into an analyzable format. The input is the video data sent from the server, which is decompressed and formatted as needed. The output is a dataset in a format usable by the analysis engine. Here, for example, consecutive frames are extracted from a JPEG image and prepared for analysis.

[0658] Step 3:

[0659] The device analyzes abnormal behavior using a prepared dataset. The input is transformed sequential frame data, and the output is the identification and location information of the detected abnormal behavior. Data processing is performed by running a motion detection algorithm using OpenCV. The device evaluates the differences in movement between frames and the length of time spent stationary, and detects deviations from the set normal behavior pattern.

[0660] Step 4:

[0661] When an anomaly is detected, the device initiates the process of notifying the relevant authorities. The input is the result of the detected abnormal behavior, and the output is the report content and a video clip. The device automatically compiles the necessary information and transmits it via email or API. Specifically, it generates a notification that includes a portion of the video, the time of occurrence, and location information.

[0662] Step 5:

[0663] The terminal notifies the mobile device of detected anomalies. Inputs are the detection results and video clips of the abnormal behavior, while output is an alert sent to the user. The terminal uses a communication protocol to send push notifications to the mobile device, allowing the user to immediately check the situation. As a concrete example, a notification appears on the user's smartphone, and they can view the on-site video through the application.

[0664] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0665] This invention relates to a system that uses video and audio data obtained from information acquisition devices such as surveillance cameras and microphones to monitor the safety of the elderly and infants in real time. This system incorporates an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection.

[0666] This system begins with a server collecting video and audio data received from an information acquisition device. The server then transmits this data to a terminal, where it is formatted into an analyzable format. The generating AI on the terminal uses the video data to detect abnormal behavior and simultaneously uses an emotion engine to analyze the user's emotions from the audio data.

[0667] The emotion engine uses emotion recognition technology to identify a user's emotional state from their tone of voice and speaking style. For example, it evaluates emotions such as whether an elderly person is feeling anxious or whether an infant is showing signs of sudden crying, and comprehensively detects signs of abnormality. If an abnormality is detected, the device uses that data to make an emergency call to the relevant authorities. If necessary, parts of the relevant video or audio will be attached to the call.

[0668] Users are immediately notified of the results of anomaly detection and emotion recognition. Through the mobile app, users can monitor the situation on-site and the evolution of emotions in real time. This feature enables faster and more appropriate responses, allowing for additional instructions and adjustments as needed.

[0669] As a concrete example, consider a scenario where an elderly person living alone suddenly expresses anxiety. In this case, the terminal analyzes video and audio data and determines that the person is experiencing anxiety. Based on this, it can send an alert to the appropriate contacts and, if necessary, dispatch medical staff. This system enables advanced anomaly detection that incorporates changes in emotions, providing a support system that allows people to live their daily lives with greater peace of mind.

[0670] The following describes the processing flow.

[0671] Step 1:

[0672] The server receives video and audio data in real time from the information acquisition device. This allows for the acquisition of visual and auditory information about the monitored target.

[0673] Step 2:

[0674] The server sends the received data to the terminal. The terminal converts this data into a parseable format and prepares for the next processing step.

[0675] Step 3:

[0676] The terminal uses a generating AI to analyze abnormal behavior based on the converted video data. The AI ​​compares the current behavior with past behavioral patterns to detect unusual actions and movements.

[0677] Step 4:

[0678] The device uses an emotion engine to analyze the user's emotions based on voice data. It analyzes the pitch and tone of the voice to evaluate emotions such as anxiety and stress.

[0679] Step 5:

[0680] The device integrates video and emotion analysis results to make a comprehensive judgment about the presence or absence of anomalies. If an anomaly is detected, it triggers the next action.

[0681] Step 6:

[0682] The terminal uses anomaly detection data to send an emergency alert to nearby relevant organizations. In addition to conventional messages, the alert can also include analyzed emotional information.

[0683] Step 7:

[0684] The device instantly sends notifications of anomaly detection and emotion recognition to the user via a mobile app. This allows the user to immediately check the situation and take action.

[0685] Step 8:

[0686] The user receives a notification, opens the app, and checks the video and audio from the scene. This allows the user to understand the situation and send out necessary instructions.

[0687] (Example 2)

[0688] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0689] In modern times, effectively monitoring the safety of the elderly and infants is a critical challenge. Traditional monitoring systems have limitations in detecting abnormal behavior and require further improvement. Furthermore, they cannot capture emotional changes, potentially missing signs of abnormality. Overcoming these challenges and providing a safer living environment is essential.

[0690] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0691] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, means for detecting abnormal behavior using a generated AI model based on the converted data, means for analyzing emotional states from audio data based on emotion recognition technology, means for notifying relevant organizations when abnormal behavior or anxiety is detected, and means for notifying relevant parties of the abnormality and allowing them to check the situation in real time on a mobile terminal. This enables comprehensive safety management that takes into account not only the detection of abnormal behavior but also changes in emotions.

[0692] An "information acquisition device" is a device that collects data such as video and audio and provides it to a server.

[0693] "Real-time" refers to a state where information is processed and analyzed as soon as it is generated, and the results are reflected immediately.

[0694] An "analyzable format" refers to a format in which collected data has been appropriately processed and can be analyzed using AI models or other tools.

[0695] A "generative AI model" is an artificial intelligence model that recognizes patterns from new data based on previously learned data and performs specific tasks.

[0696] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns and is an important indicator that should be monitored.

[0697] "Emotion recognition technology" refers to technology that identifies a person's emotional state based on voice and text data.

[0698] "Reporting" refers to the act of informing the relevant authorities about information regarding detected anomalies.

[0699] A "mobile device" is a device that a user can carry with them and is used to receive information and perform operations via applications.

[0700] "Related organizations" refers to public or private organizations that provide support or response when an anomaly occurs.

[0701] Embodiments of this invention include the following systems and functions.

[0702] The system is initially composed of a server that receives video and audio data in real time using information acquisition devices such as surveillance cameras and microphones. The server receives the data via a communication interface and immediately transmits it to terminals. The terminals use a data conversion module to process the received data into a format that can be analyzed.

[0703] The device is equipped with a generative AI model for analyzing video data, enabling the detection of abnormal behavior. Furthermore, emotion recognition technology is used for analyzing audio data. This technology analyzes the tone and speed of speech to identify the user's emotional state. This enables advanced safety management that captures not only abnormal movements but also emotional changes.

[0704] If an anomaly is detected, the device automatically notifies the relevant authorities. The notification includes data on the detected abnormal behavior and emotional changes. Furthermore, the user is immediately notified of the anomaly, and this information can be viewed on their mobile device. In this way, the user can understand the situation in real time and take necessary actions quickly.

[0705] As a concrete example, in a living environment with an elderly person living alone, this system could detect an increase in the elderly person's anxiety and promptly notify care services or medical institutions. This would allow the user to live their daily life with greater peace of mind. An example of a prompt message would be, "Detect anxiety from the elderly person's tone of voice and suggest appropriate measures."

[0706] Ultimately, this system aims to leverage its advanced data processing capabilities to support users' lives while ensuring safety.

[0707] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0708] Step 1:

[0709] The server receives video and audio data in real time from surveillance cameras and microphones. The input data consists of stream data from the cameras and microphones, which is acquired by a network module within the server. The output is the collected raw video and audio data. Specifically, the server uses a communication protocol to ensure uninterrupted data reception.

[0710] Step 2:

[0711] The server sends the received data to the terminal. The input is the raw data mentioned earlier, and a communication format conversion is performed to convert it into a format suitable for transmission. The output is data packets grouped into an appropriate batch size. In actual operation, the server uses its data transmission function to ensure stable data transfer.

[0712] Step 3:

[0713] The terminal formats the received data into a format that can be analyzed. The input is data packets sent from the server, which are processed by the data conversion module. The output is data formatted for analysis, such as video frames and audio clips. Specifically, the terminal performs frame extraction and noise reduction before analyzing the data.

[0714] Step 4:

[0715] The on-device AI detects abnormal behavior. The input is formatted video data, and the AI ​​model is executed. The output is data of the detected abnormal behavior. Specifically, the AI ​​model analyzes the movement pattern for each frame and captures movements that deviate from normal movement patterns.

[0716] Step 5:

[0717] The device uses an emotion engine to analyze emotional states using voice data. The input is formatted voice data to which emotion recognition technology is applied. The output is information on the analyzed emotional state. Specifically, the model extracts embedded features such as voice tone and speech rate to estimate the emotional state.

[0718] Step 6:

[0719] The device will notify the relevant authorities if it detects abnormal behavior or anxiety. The input is the data on abnormalities and emotions obtained in the previous step, and a notification message is generated based on this data. The output is an emergency alert sent to the designated contact. Specifically, the message generation program attaches the relevant data to ensure rapid notification.

[0720] Step 7:

[0721] Users receive anomaly notifications and check the real-time status on their mobile devices. The input is notification data sent from the device, which allows for understanding the current situation. The output is a notification display and an interactive interface that the user can view and operate. Specifically, the mobile app displays notifications as pop-ups and provides functions to visualize video and sentiment analysis results.

[0722] (Application Example 2)

[0723] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] To ensure the safety of the elderly and infants in real time, it is necessary not only to detect abnormal behavior but also to quickly capture changes in their emotions. However, conventional technology is limited to detecting abnormal behavior and has the drawback of not being able to detect early signs of abnormality through emotion analysis from voice.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0726] In this invention, the server includes means for receiving video and audio data acquired from an information acquisition device in real time, means for converting the received data into an analyzable format, and means for detecting abnormal behavior from the converted data and analyzing emotional states from the audio data. This enables rapid and appropriate responses through highly accurate anomaly detection and emotion analysis.

[0727] An "information acquisition device" is a device that has the function of collecting video data and audio data.

[0728] "Video data" refers to image information acquired by surveillance cameras and other video devices.

[0729] "Audio data" refers to sound information acquired by a microphone or similar device.

[0730] A "means of receiving data in real time" refers to a mechanism that processes acquired data almost simultaneously.

[0731] "Means of converting to an analyzable format" refers to a function that processes received data into a format that can be processed by an analysis engine.

[0732] An "analysis means for detecting abnormal behavior" is a mechanism that analyzes video data to identify deviations from normal behavior.

[0733] A "means for analyzing emotional states" refers to a mechanism that analyzes voice data to identify the user's emotions.

[0734] "A means of notifying mobile devices and reporting to relevant organizations" refers to a system that transmits information to portable communication devices or relevant departments when an anomaly is detected.

[0735] "Means for transmitting video and audio of abnormal behavior to relevant organizations" refers to a function that transmits data that serves as evidence of detected abnormalities to public or designated organizations.

[0736] The system necessary to implement this invention consists of an information acquisition device, a server, a terminal, and a mobile device used by the user. The server receives video and audio data acquired from surveillance cameras and microphones in real time. The received data is converted into an analyzable format, and analysis is performed on the server to detect abnormal behavior. Video analysis libraries such as OpenCV are used for this analysis.

[0737] Furthermore, voice analysis technology is applied to the analysis of voice data to identify the user's emotional state from their tone of voice and speaking style. For example, by using voice recognition services such as Azure Cognitive Services, changes in emotion can be analyzed in real time. This combination of emotion recognition and abnormal behavior detection enables highly accurate safety monitoring.

[0738] The user's device is immediately notified of any detected anomalies. Push notifications are sent via Firebase Cloud Messaging, allowing the user to take appropriate action based on the notification. If necessary, the relevant authorities will also be notified. In such cases, video and audio recordings capturing abnormal behavior or emotional changes may be attached.

[0739] For example, if an elderly person begins to feel anxious at home, the system analyzes their emotions from their voice and detects it as an abnormality. This immediately sends a notification to the family member, allowing for a quick response while viewing the video and audio. This system provides a safe and secure monitoring environment.

[0740] An example of a prompt for using a generative AI model to more precisely analyze abnormal behavior would be: "Describe a system that monitors the daily activities of elderly people, detects emotional changes, and provides real-time notifications." This allows the AI ​​model to perform a detailed analysis and enable flexible responses.

[0741] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0742] Step 1:

[0743] The server receives video and audio data in real time from surveillance cameras and microphones. The input is raw data from the cameras and microphones, while the output is data temporarily stored within the server. This data is processed as a stream to maintain real-time performance.

[0744] Step 2:

[0745] The server converts the received raw data into a format that can be analyzed. The input is raw video and audio data, and the output is data in a format that the analysis engine can process. The OpenCV library is used to convert the video data to a standard image format, and the audio is converted to WAV or MP3 format.

[0746] Step 3:

[0747] The server detects abnormal behavior based on the converted video data. The input is the converted video data, and the output is the evaluation result regarding the abnormal behavior. OpenCV is used to analyze the movement and compare it with predetermined abnormal behavior patterns.

[0748] Step 4:

[0749] The server analyzes audio data to identify emotional states. The input is the converted audio data, and the output is the detected emotion label. Speech recognition technologies such as Azure Cognitive Services are used to analyze emotions from voice tone and speed.

[0750] Step 5:

[0751] The server determines whether an anomaly has occurred based on the detection results and outputs the result. The input is the evaluation result of abnormal behavior and the result of sentiment analysis, and the output is a flag indicating the presence or absence of an anomaly and its detailed information. If an anomaly is detected, the server prepares to send a notification request to the mobile device.

[0752] Step 6:

[0753] The server pushes information about the anomaly to the user's mobile device. The input is detailed information about the anomaly, and the output is a notification message to the user. Firebase Cloud Messaging is used for rapid notification.

[0754] Step 7:

[0755] Users receive notifications and view detailed video and audio via their mobile devices. The input is the notification message sent from the server, and the output is the video and audio content of the anomaly displayed on the device. This allows for rapid countermeasures to be taken.

[0756] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0757] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0758] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0759] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0760] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0761] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0762] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0763] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0764] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0765] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0766] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0767] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0768] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0769] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0770] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0771] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0772] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0773] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0774] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0775] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0776] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0777] The following is further disclosed regarding the embodiments described above.

[0778] (Claim 1)

[0779] A means for receiving video data acquired from an information acquisition device in real time,

[0780] A means for converting received video data into an analyzable format,

[0781] An analysis means for detecting abnormal behavior based on converted video data,

[0782] A means of notifying relevant authorities when abnormal behavior is detected,

[0783] Means of notifying relevant parties of an anomaly,

[0784] A system that includes this.

[0785] (Claim 2)

[0786] The system according to claim 1, wherein the analysis means uses deviations from normal behavior patterns based on learning data as a criterion for judgment.

[0787] (Claim 3)

[0788] The system according to claim 1, wherein the reporting means transmits the abnormal behavior to the relevant organization along with a video of the abnormal behavior.

[0789] "Example 1"

[0790] (Claim 1)

[0791] A device that receives video data acquired from an information acquisition device in real time, and transmits it to a data processing device,

[0792] A device that converts received video data into an analyzable data format, enabling analysis in the next stage,

[0793] An analysis device that detects abnormal behavior from data transformed using a specific algorithm,

[0794] A communication device that notifies the relevant authorities when abnormal behavior is detected,

[0795] A notification device that alerts registered parties to abnormalities,

[0796] A system that includes this.

[0797] (Claim 2)

[0798] The system according to claim 1, wherein the analysis device uses deviations from normal behavior patterns based on stored learning data as a criterion for judgment.

[0799] (Claim 3)

[0800] The system according to claim 1, wherein the communication device transmits information to the relevant organization, attaching a portion of the video footage of the abnormal behavior.

[0801] "Application Example 1"

[0802] (Claim 1)

[0803] A means for receiving video information acquired from an information acquisition means in real time,

[0804] A means for converting received video information into an analyzable information format,

[0805] An analysis means for detecting abnormal behavior based on the converted video information,

[0806] A means of notifying relevant agencies when abnormal behavior is detected,

[0807] A means of notifying relevant parties of the anomaly,

[0808] A means of notifying a mobile device when abnormal behavior is detected,

[0809] A system that includes this.

[0810] (Claim 2)

[0811] The system according to claim 1, wherein the analysis means uses deviations from normal behavior patterns based on learning data as a criterion for judgment, and provides a means for the user to confirm the situation and respond via the mobile information terminal.

[0812] (Claim 3)

[0813] The system according to claim 1, wherein the notification means transmits video footage of abnormal behavior to the relevant organization and provides the user with an image of the situation via the mobile information terminal, thereby enabling a rapid response.

[0814] "Example 2 of combining an emotion engine"

[0815] (Claim 1)

[0816] A means for receiving video and audio data acquired from an information acquisition device in real time,

[0817] A means of converting received data into a format that can be parsed,

[0818] Based on the converted data, an analysis means for detecting abnormal behavior using a generative AI model,

[0819] A means of analyzing emotional states from voice data based on emotion recognition technology,

[0820] A means of reporting to the relevant authorities when abnormal behavior or anxiety is detected,

[0821] A means of notifying relevant parties of an anomaly and allowing them to check the real-time situation on their mobile devices,

[0822] A system that includes this.

[0823] (Claim 2)

[0824] The system according to claim 1, wherein the analysis means uses deviations from normal behavioral patterns and changes in emotional state based on learning data as criteria for judgment.

[0825] (Claim 3)

[0826] The system according to claim 1, wherein the notification means transmits data based on the results of abnormal behavior and emotion analysis to the relevant organizations.

[0827] "Application example 2 when combining with an emotional engine"

[0828] (Claim 1)

[0829] A means for receiving video data and audio data acquired from an information acquisition device in real time,

[0830] A means for converting received video data and audio data into an analyzable format,

[0831] An analysis means for detecting abnormal behavior based on converted video data,

[0832] A method for analyzing emotional states from audio data and detecting signs of abnormalities,

[0833] A means of notifying mobile devices and reporting to relevant organizations when an anomaly is detected,

[0834] A means of transmitting video and audio of abnormal behavior to relevant organizations,

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The system according to claim 1, wherein the analysis means uses deviations from normal behavioral patterns and emotional states from voices based on learning data as criteria for judgment.

[0838] (Claim 3)

[0839] The system according to claim 1, wherein the reporting means transmits the abnormal behavior to the relevant organization along with video and audio recordings. [Explanation of Symbols]

[0840] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving video data acquired from an information acquisition device in real time, A means for converting received video data into an analyzable format, An analysis means for detecting abnormal behavior based on converted video data, A means of notifying relevant authorities when abnormal behavior is detected, Means of notifying relevant parties of an anomaly, A system that includes this.

2. The system according to claim 1, wherein the analysis means uses deviations from normal behavior patterns based on learning data as a criterion for judgment.

3. The system according to claim 1, wherein the reporting means transmits the abnormal behavior to the relevant organization along with a video of the abnormal behavior.

Images