system

Abstract

We provide the system. [Solution] An analytical means that receives audio and video data of visitors in real time and analyzes them to identify individuals, An identification means for identifying a person identified by the aforementioned analysis means as a suspicious person if that person is not registered in advance, A warning issuing mechanism that issues a warning based on the result of identifying a person as suspicious, Information sharing means for sharing surrounding information via a regional network, A system that includes this.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the increase in the elderly population, crime prevention and safety assurance for the elderly living alone have become important issues. However, conventional crime prevention systems have problems such as insufficient detection of suspicious persons and interactive dialogue with visitors, making it difficult for the elderly and their families to live with peace of mind. In addition, the utilization of local networks for prompt response in emergencies has not been effectively carried out. There is a demand for the provision of a new crime prevention support system that solves these problems.

Means for Solving the Problems

[0005] This invention includes an analysis means that analyzes visitor audio and video data in real time to identify individuals. Furthermore, based on the analysis results, it identifies unregistered individuals as suspicious and immediately issues a warning, thereby improving crime prevention awareness. This system can also share surrounding information via a local network, enhancing crime prevention effectiveness throughout the community. In addition, by analyzing visitor behavior patterns and detecting abnormal behavior, a faster and more accurate response can be achieved. Through these means, the aim is to provide a safe and secure living environment for the elderly.

[0006] "Analysis means" refers to technical means for receiving audio and video data in real time and identifying visitors.

[0007] "Identification means" refers to a means that determines whether a person identified by the analysis means is registered, and if not registered, identifies them as a suspicious person.

[0008] A "warning system" is a means of quickly issuing a warning when a person is identified as a suspicious individual, and notifying the user of the presence of such a person.

[0009] "Information sharing methods" refer to means of sharing crime prevention information in the surrounding area using a local network, thereby enhancing the overall crime prevention effectiveness of the community.

[0010] "Behavioral analysis means" refers to a method for analyzing behavioral patterns based on visitors' audio and video data to detect abnormal behavior.

[0011] "Communication means" refers to a technical interface that enables remote interaction with visitors and facilitates real-time communication. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] The crime prevention support system according to the present invention is a technology that uses visitor voice and video data to perform real-time analysis in order to ensure the safety of the elderly. This system consists of a server, terminals, and users, each performing the following functions.

[0034] The server receives audio and video data from sensors and performs analysis to identify visitors. Specifically, it uses a machine learning model to recognize visitors' faces and compares them with information registered in a database to determine whether the visitor is a pre-authorized individual. If an unregistered visitor is detected during this process, it is identified as a suspicious person, and information is generated to issue a warning.

[0035] The terminal receives warning information sent from the server and issues warnings to the user. The terminal uses push notifications and voice alarms to alert elderly individuals to the presence of suspicious persons at the appropriate time. Furthermore, it has a function that enables remote communication with visitors based on security camera data, allowing elderly individuals to check the visitor's situation in real time.

[0036] Users can receive warnings and notifications via their devices and communicate securely with external visitors. This reduces anxiety in daily life and ensures safety at home. If necessary, users can report the situation to the local crime prevention network, contributing to broader crime prevention efforts.

[0037] For example, if an unfamiliar visitor C appears at an elderly person's home one day, the server immediately analyzes the audio and video data and performs facial recognition. If the result reveals that visitor C is an unregistered person, they are identified as a suspicious person, and a warning is sent to the elderly person via their device. Upon receiving this information, the elderly person can use the device's camera system to interact with visitor C and confirm the situation.

[0038] This allows elderly people to live their daily lives with peace of mind and enables prompt responses as needed. This system will make a significant contribution to ensuring safety in today's aging society.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server receives audio and video data from the sensors in real time. This data is used as input for analysis necessary to identify visitors.

[0042] Step 2:

[0043] The server uses the received video data to apply a machine learning model and recognize the visitor's face. This process extracts facial features from the video and compares them with an existing database.

[0044] Step 3:

[0045] The server checks the analysis results and determines whether the identified face is registered in the database. If the face is not registered, it is identified as a suspicious person and suspicious person information is generated.

[0046] Step 4:

[0047] The server analyzes the audio data and detects unusual sounds (e.g., loud noises or shouting). Based on these analysis results, it sets triggers for issuing warnings.

[0048] Step 5:

[0049] The device receives warning information sent from the server and sends a notification to the user. This notification is provided as a push notification or an audio alarm.

[0050] Step 6:

[0051] The device activates a communication mechanism for remotely interacting with visitors. This allows the user to check the visitor's status via the camera.

[0052] Step 7:

[0053] Based on the warnings received through the device, the user checks the visitor's status and determines the necessary action. Only after confirming the visitor's safety will the door be unlocked.

[0054] Step 8:

[0055] The server shares surrounding information with the local crime prevention network and takes the initiative as part of crime prevention measures for the entire region.

[0056] Step 9:

[0057] Users can improve crime prevention by reporting warning information to their local community as needed and strengthening cooperation.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] In modern society, ensuring the safety of elderly people in their homes is a critical issue. It is necessary to quickly and accurately identify visitors and prevent intruders from entering homes. However, conventional systems struggle with real-time responses, posing challenges to information sharing within the community and the rapid issuance of warnings.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes processing means for receiving and processing audio and video data in real time, identification means for identifying visitors identified by the processing means as unidentified individuals if they are not pre-registered, and notification means for sending a warning based on the result of identifying an individual as unidentified. This enables elderly people to quickly receive information about suspicious individuals and to implement effective crime prevention measures in cooperation with local networks.

[0063] "Audio data" refers to digitized data that includes visitors' voice information and is used for visitor identification and behavioral analysis.

[0064] "Video data" refers to digital video containing the visitor's visual information, and is image data used for facial recognition and motion analysis.

[0065] "Processing means" refers to computing devices and programs used to analyze acoustic and video data to identify visitors and analyze their behavior.

[0066] A "discrimination means" is a device that has the function of determining whether a visitor is a person who has been registered in advance, based on the analyzed acoustic and video data.

[0067] "Notification means" refers to devices or systems that promptly transmit information about identified unidentified individuals as a warning to elderly people and relevant organizations.

[0068] "Communication means" refers to communication technologies or equipment used to enable remote interaction with visitors.

[0069] "Information sharing means" refers to networks and technologies for sharing information about visitors with other stakeholders and organizations in order to improve safety within a region.

[0070] This invention provides a security system that receives and analyzes visitors' audio and video data in real time to ensure the safety of the elderly.

[0071] Specifically, the server receives audio and video data from audio and video sensors installed at entrances and other locations. The received data is analyzed using an image processing library (e.g., OpenCV) and a machine learning framework (e.g., TENSORFLOW®). Through this analysis, the server performs facial recognition of visitors and determines their permission status by comparing them with pre-registered information stored in a database. If an unregistered visitor is detected, the server identifies them as a suspicious person, generates warning information, and sends it to the terminal.

[0072] The terminal is responsible for transmitting warning information sent from the server to the elderly. The terminal consists of smartphones and tablets, and promptly displays warnings via push notification services (e.g., Firebase Cloud Messaging). Furthermore, the terminal allows the user to interact remotely with visitors via its built-in camera and microphone. This interaction function enables the elderly to check the visitor's situation and take appropriate action as needed.

[0073] Users make decisions to ensure the safety of their homes based on warnings and notifications received via their devices. Users can verify the faces and voices of visitors or report the situation to the local security network using an application on their device. This process strengthens coordinated crime prevention measures across the entire community.

[0074] For example, if a suspicious person approaches the front door of an elderly person's home, the server immediately analyzes the data and sends a warning to the terminal. The user can view the video of the suspicious person displayed on the terminal screen and confirm the situation using the remote dialogue function. In this way, real-time vigilance and information sharing within the community are achieved.

[0075] An example of a prompt message would be, "Please tell me about the specific operation and dialogue functions of the security system developed for the elderly."

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The server receives audio and video data in real time from acoustic and video sensors. Inputs include visitor audio and video. This data is securely transmitted to the server using security protocols. Outputs include the received raw data, ready for use in the next analysis step.

[0079] Step 2:

[0080] The server analyzes the received audio and video data. During this process, it uses a facial recognition algorithm to detect visitors' faces and compares them to facial images registered in a database. The input is raw audio and video data, and the output is specific facial features and audio information for person identification. Specifically, a Python script uses the OpenCV library to perform face detection.

[0081] Step 3:

[0082] The server determines whether a visitor is a pre-registered or unregistered person based on the analysis results. The input is the visitor's feature data obtained in the previous step, and the output is the determination of whether or not the person is authorized. In this process, a machine learning model is used to match the feature vectors.

[0083] Step 4:

[0084] If the server identifies a person as suspicious based on the identification result, it generates a warning message and sends it to the terminal. In this case, the input is the identification result, and the output is a warning message containing information about the suspicious person. Specifically, the message is generated in JSON format and sent via the network.

[0085] Step 5:

[0086] The terminal receives warning messages from the server and notifies the user of the warning. The input is the warning message sent from the server, and the output is the warning information displayed on the terminal screen and an audio alarm. Specifically, the terminal's notification function is used to generate push notifications and audio alerts to immediately deliver warnings to elderly users.

[0087] Step 6:

[0088] The user interacts remotely with visitors via their device. The input is real-time data from the camera and microphone, while the output is video and audio calls displayed on the device screen. Specifically, the user communicates with visitors through a video chat application, checks the visitor's status, and makes decisions to ensure their safety.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] To ensure the safety of the elderly and residents, it is necessary to effectively monitor visitors coming and going and to take immediate action if unregistered individuals or suspicious behavior are detected. However, current security systems struggle to identify individuals and analyze abnormal behavior in real time, and they do not adequately provide immediate notification and countermeasures to facility staff and residents. Furthermore, the lack of secure means of communication with visitors can lead to delays in confirming their situation and responding to such situations.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if the person is not registered in advance; and a warning issuing means that issues a warning based on the result of identifying the person as a suspicious person. This enhances visitor monitoring within the facility and enables the provision of prompt and appropriate warnings and information to staff and residents.

[0094] "Analysis means" refers to technology that receives visitor audio and video data in real time and uses that data to identify individuals.

[0095] "Identification means" refers to a method of identifying a person as a suspicious person when that person has not been previously registered by the analysis means.

[0096] A "warning system" is a method for issuing warnings to users or facility staff based on the results of identifying someone as a suspicious person.

[0097] "Means of disseminating information" refers to methods for transmitting information necessary to ensure the safety of residents in the surrounding environment to relevant parties.

[0098] "Communication methods" refer to technologies that enable remote interaction with visitors using images and audio, thereby ensuring secure communication.

[0099] "Behavioral analysis means" refers to a processing method for analyzing visitor behavior patterns based on audio and video data and detecting abnormal behavior.

[0100] "Methods for sending push notifications" refer to methods for evaluating the safety of visitors within a facility based on analyzed data and immediately notifying staff and relevant parties as needed.

[0101] The system that realizes this invention mainly consists of a server, a terminal, and a user. The server is responsible for collecting visitor audio and video data in real time. Specifically, it acquires data through the camera and microphone of a smartphone or a fixed device. For data analysis, machine learning frameworks such as TensorFlow Lite and facial recognition technology using OpenCV are used. WebRTC and the like are used for audio analysis.

[0102] The server uses analysis tools to identify visitors based on the acquired data. If the identified visitor is not previously registered in the database, the identification tool recognizes them as a suspicious person, and an immediate warning system is activated. This sends a warning to the terminal. The terminal is a smartphone or mobile device, and the system sends push notifications or voice alerts to inform the user of the emergency.

[0103] This system also includes communication methods that enable secure remote interaction with visitors. Users can interact with visitors in real time via video and audio through their devices. This feature allows users to check on visitors' situations and intervene as needed.

[0104] As a concrete example, staff working at a nursing home use smartphones with the "Safe Visitor Check" app installed, and receive an immediate alert if an unregistered visitor appears. The staff can then interact with the visitor through the device to confirm their safety. An example of a prompt message to the generated AI model in this system is, "Please compare the new visitor's face with the database. If it cannot be recognized, please issue an alert."

[0105] This invention can significantly improve safety within the facility and ensure the peace of mind and security of residents.

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The server receives audio and video data from visitors in real time via smartphones and cameras. Input is video and audio obtained through the camera and microphone, and output is data transferred to the server. To receive this data, the device needs to be connected to a Wi-Fi or mobile network.

[0109] Step 2:

[0110] The server performs face recognition using the received video data. The input is the video data acquired in step 1, and the data is processed using OpenCV. The output is facial feature information extracted from the video data. The facial features are identified by a machine learning algorithm and compared with a registered database.

[0111] Step 3:

[0112] The server analyzes the audio data to identify the visitor's voiceprint. The input is the audio data obtained in step 1, and the output is the estimated voiceprint. Using WebRTC-based audio processing technology, the server analyzes the audio data and evaluates voice matching by comparing it with registered voices.

[0113] Step 4:

[0114] The server determines whether the visitor is registered or not based on the analysis results. The input is the output results from steps 2 and 3. The output is a flag indicating whether the visitor is registered or not. If the visitor is not registered, they are identified as a suspicious person.

[0115] Step 5:

[0116] The terminal receives notifications from the server and issues warnings. The input is a suspicious person identification notification from step 4, and the output is a push notification and voice alert to the elderly or staff. Notifications are primarily made using the terminal's notification system.

[0117] Step 6:

[0118] Users interact with visitors in real time via image and audio through their devices. Input is real-time video and audio of the visitor, and output is information obtained through communication with the visitor. A secure communication channel is opened through the device, allowing users to monitor the situation.

[0119] Step 7:

[0120] The server analyzes visitor behavior patterns and detects anomalies. Input is continuous audio and video data from step 1, and output is a notification indicating whether the behavior is normal or abnormal. A generative AI model is used for behavior analysis, and in the event of an anomaly, a prompt example message such as "Analyze the behavior pattern and detect anomalies" is sent.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] The crime prevention support system according to the present invention provides advanced technology that not only analyzes visitor voice and video data in real time but also monitors the user's emotional state by combining it with an emotion engine. This system consists of a server, a terminal, and a user, each playing the following roles.

[0123] The server receives audio and video data from sensors and performs visitor facial recognition and behavioral analysis. The server also uses an emotion engine to analyze the user's emotions and collect data related to their emotional state. This analysis helps determine if the user is experiencing stress or anxiety and prepares to issue situation-appropriate warnings.

[0124] The device receives warning information sent from the server and provides notifications to the user. This process includes push notifications and voice alarms. It also has a feature that automatically sends an external alert if the emotion engine determines the user is in an emergency stress state. In this way, the user can quickly receive the necessary support and intervention. Furthermore, it includes communication capabilities that enable remote interaction with visitors, providing an environment where the user can see visitors in real time.

[0125] Users respond to visitors based on notifications from their device. Furthermore, their emotional state is monitored through an emotion engine built into the device, improving psychological safety when they are alone. For example, if visitor D appears at the front door, the server immediately analyzes the visitor's data, evaluating facial recognition and behavioral patterns. Simultaneously, if the emotion engine detects signs of stress from the user's voice, the device sends a notification to the user encouraging relaxation and, if necessary, sends an alert to the local network or emergency contacts.

[0126] These features allow elderly people to maintain a more secure living environment. This system provides comprehensive crime prevention measures that take emotional states into consideration, contributing particularly to improved safety and security for the elderly.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The server receives audio and video data from visitors in real time from sensors. Audio data is obtained from microphones, and video data from surveillance cameras.

[0130] Step 2:

[0131] The server analyzes the received video data and uses a machine learning model to recognize the visitor's face. Features extracted from the video are compared with existing data in the database to determine if they are already registered.

[0132] Step 3:

[0133] If an unregistered person is identified, the server identifies that person as a suspicious individual and generates suspicious person information. This information is transmitted to the terminal via a warning system.

[0134] Step 4:

[0135] The server then analyzes the visitor's voice data. This voice analysis detects unusual sounds and unusual voice tones, and the results are used for behavioral analysis.

[0136] Step 5:

[0137] The server uses an emotion engine to analyze the user's voice data and evaluate the user's emotional state. In this process, it analyzes changes in voice patterns and tone to determine the degree of stress and anxiety.

[0138] Step 6:

[0139] The device receives warnings from the server and user sentiment rating information, and sends appropriate notifications to the user. These notifications may include warnings about the presence of suspicious individuals or the user's stress level.

[0140] Step 7:

[0141] Users check warning notifications from their devices and take appropriate action based on the situation. They can also view real-time video footage of visitors and, if safety is confirmed, respond using the remote interaction function.

[0142] Step 8:

[0143] The device automatically sends an alert to an external emergency contact or local security network if it determines that the user is in an emergency stress state.

[0144] Step 9:

[0145] Users can share crime prevention information with their local community and family in conjunction with the alerts they receive, promoting collaborative crime prevention measures. This collaboration improves the overall safety of the community.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] In modern society, crime prevention is a critical issue, and there is a particular need to strengthen security measures at home. However, there is a lack of systems that provide comprehensive measures that take into account visitor identification and the emotional state of users, making it difficult to quickly and accurately identify suspicious individuals and take appropriate action. To solve this problem, it is necessary to analyze visitor data in real time and monitor the psychological state of users.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes data acquisition means for receiving audio and video data of visitors in real time; analysis means for analyzing the data collected by the data acquisition means to identify individuals; identification means for identifying individuals not previously registered as suspicious based on the results of the person identification; emotion analysis means for analyzing the user's emotional state using the analysis means and evaluating stress and anxiety; warning issuing means for issuing warnings according to the results of identification as a suspicious person and the user's emotional state; information sharing means for sharing surrounding information via a local network; and notification means for providing notifications to the user in a visible manner. This enables a high level of crime prevention effectiveness by integrating visitor identification and user emotion monitoring.

[0151] "Data acquisition means" refers to the components of devices and systems for receiving visitors' audio and video data in real time.

[0152] "Analysis means" refers to a function that processes audio and video data collected by data acquisition means to identify visitors.

[0153] "Identification means" refers to a function that identifies unregistered individuals as suspicious persons based on personal information identified by analysis means.

[0154] "Emotional analysis means" refers to a function that analyzes the user's voice data and evaluates the emotional state contained within it.

[0155] The "warning notification system" is a function that generates and sends a warning based on the identification results of a suspicious person and the user's emotional state.

[0156] "Information sharing means" refers to functions for sharing relevant information with other systems and users via a regional network.

[0157] A "notification method" is a function that delivers visual or auditory warnings or information to the user.

[0158] The security support system of this invention relies on cooperation between a server, a terminal, and a user. First, the server uses highly sensitive sensors and cameras to acquire visitor audio and video data in real time. Specifically, the software used can include general image analysis programs or open-source image processing libraries for face recognition, and a speech recognition engine for audio analysis. For example, OpenCV can be used for image analysis, and the speech recognition engine's API can be used for audio analysis.

[0159] The server analyzes the acquired data to identify visitors and matches them against an existing database using facial recognition technology. The server also analyzes visitor behavior patterns to detect unusual behavior. Machine learning algorithms can be used for this analysis. Furthermore, the server performs sentiment analysis based on the user's voice data to evaluate the user's emotional state.

[0160] The terminal is responsible for notifying the user of information and warnings sent from the server. It uses devices such as smartphones, tablets, and smart speakers as notification methods. The terminal also has a function that, based on the results of emotion analysis, displays a message encouraging relaxation on the screen if it determines that the user is experiencing stress.

[0161] Users are expected to take action in response to notifications from their devices. For example, if a visitor is identified as suspicious, the user can check the alert on their device and take appropriate action immediately. They can also use remote interaction features to communicate with visitors through their device as needed.

[0162] For example, if the system analyzes that "Visitor D is exhibiting suspicious behavior," the terminal will notify the user with the message, "This person may be suspicious. Please ensure your safety." Based on this information, the user can take appropriate action in response to the alert. Furthermore, by inputting a prompt such as, "Please tell me how to improve the user's emotional state," into the generating AI model, the system can learn more efficient countermeasures.

[0163] This allows the system to provide comprehensive security measures that take into account not only the behavior of visitors but also the psychological state of the users. It is particularly beneficial in improving the safety and security of the elderly.

[0164] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0165] Step 1:

[0166] The server acquires audio and video data of visitors in real time from sensors and cameras. This input data includes the visitor's voice characteristics and facial movements. After receiving this data, the server converts it into an appropriate format and prepares it for analysis. Specifically, it digitally processes the audio data to extract features and divides the video data into frames.

[0167] Step 2:

[0168] The server analyzes the acquired audio and video data. For audio data, a speech recognition engine is used to convert speech to text and extract the content of the visitor's speech. For video data, image analysis software is used to recognize the visitor's face and compare it with the registered database. Audio and video are given as input, and the output is the result of person identification. For example, it can determine whether the visitor's face is already registered.

[0169] Step 3:

[0170] The server analyzes visitor behavior patterns based on the analyzed data. It uses machine learning models to perform data calculations that detect patterns that differ from normal behavior. If the analysis results match the patterns of a suspicious person, the server uses identification methods to confirm this. Behavioral data is provided as input, and the output is a result of the behavioral anomaly detection.

[0171] Step 4:

[0172] The server analyzes the user's voice data to assess their emotional state. This is done by extracting and classifying emotional expressions in the voice using emotion analysis software. Based on this analysis, the server evaluates whether the user is experiencing stress or anxiety, and identifies potential alarm triggers. The input is the user's voice data, and the output is the result of the emotional state evaluation.

[0173] Step 5:

[0174] The device receives warning information sent from the server and notifies the user. This notification may include push notifications or audio alarms on a smartphone. The input is the warning information packet data, and the output is implemented as a visual or auditory notification to the user. The user then decides on an action regarding the visitor based on this notification.

[0175] Step 6:

[0176] The user interacts with visitors based on notifications provided by the device. If necessary, they use the device's communication methods to directly communicate with visitors and verify their safety. Once this operation is complete, the system-wide log is updated, and data is accumulated for later analysis. The input is the notification from the device, and the output is the specific action taken by the user.

[0177] (Application Example 2)

[0178] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0179] In modern society, ensuring safety during interactions with visitors and maintaining users' psychological well-being are crucial issues. This requires not only identifying suspicious individuals and analyzing visitor behavior patterns, but also systems that provide real-time dialogue with visitors and offer psychological support by analyzing users' emotional states. For elderly users and those living alone, enhancing psychological safety and reducing anxiety and stress is particularly important, making the development of comprehensive security and psychological support systems a critical challenge.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0181] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if they are not registered in advance; and a psychological analysis means that analyzes the user's emotional state from the audio and video data to detect a specific psychological state. This enables a rapid response to suspicious individuals, as well as the provision of notifications and support appropriate to the user's emotional state, thereby improving safety and a sense of security.

[0182] "Analysis means" refers to a device or program that receives visitor audio and video data in real time and processes them to identify the person.

[0183] "Identification means" refers to a device or program for recognizing a person identified by analysis means as a suspicious person when that person has not been registered in advance.

[0184] A "warning signaling device" is a device or program that issues an alarm and notifies the surrounding area based on the result of identifying a person as suspicious.

[0185] An "information sharing means" is a device or program for sharing acquired information with the surrounding area via a regional network.

[0186] "Psychological analysis means" refers to a device or program for analyzing a user's emotional state from audio and video data and detecting a specific psychological state.

[0187] A "measure transmission means" is a device or program for transmitting a corresponding measure when a specific psychological state is detected.

[0188] The system implementing this invention consists of a server, a terminal, and a user. The server receives audio and video data of visitors collected in real time via sensor devices such as cameras and microphones, and uses a face recognition API and an audio analysis API as analysis means to identify individuals and perform behavioral analysis. Furthermore, by utilizing a psychological analysis means using an emotion engine to evaluate the user's emotional state, it is possible to detect specific psychological states.

[0189] The device receives warning information transmitted from the server and provides warnings to the user via push notifications. Furthermore, if psychological analysis determines that the user is experiencing a specific psychological state, such as stress, it sends a notification to encourage relaxation. Additionally, it can automatically send alerts to emergency contacts designated by the user, if necessary.

[0190] Users can respond appropriately to visitors based on notifications provided through their devices. Furthermore, the device's built-in psychological analysis capabilities monitor the user's emotional state, enhancing their psychological safety when alone. For example, if a visitor appears at the front door, the server immediately analyzes the visitor's data and, if necessary, sends a notification to the user encouraging them to relax.

[0191] A concrete example is a scenario where, while a user is alone at home, the system monitors visitors and provides the user with a sense of security. By utilizing a generative AI model, the system can provide the user with information through prompt messages such as, "A sentiment-analyzing security app connected to your home security system will identify visitors and send real-time notifications based on their behavior."

[0192] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0193] Step 1:

[0194] The server receives visitor audio and video data in real time via the camera and microphone. This supplies the server with audio and video streams as data input. The data is then sent to the Face Recognition API and Audio Analysis API to prepare it for image and audio analysis.

[0195] Step 2:

[0196] The server analyzes the input audio and video data to identify visitors and analyze their behavior patterns. It uses a face recognition API to identify individuals and an audio analysis API to analyze the tone and content of visitors' voices. If the analysis identifies suspicious behavior or unregistered visitors, this information is output as identification data.

[0197] Step 3:

[0198] Based on the identification information output by the server, it detects an anomaly and sends a warning to the terminal as a means of issuing a warning. Specifically, it generates and sends a warning message to the user's terminal via a push notification platform (such as Firebase). The input is the identification information, and the output is a notification on the terminal.

[0199] Step 4:

[0200] The device receives a warning from the server and displays a notification to the user. The user is notified of the anomaly in real time via push notifications or audio alarms. The input is a warning message from the server, and the output is a visual and auditory notification to the user.

[0201] Step 5:

[0202] The server uses an emotion engine to analyze the user's voice data and evaluate their emotional state. If a specific psychological state (e.g., stress) is detected by the psychological analysis tool, the evaluation result is output. The input is the user's voice data, and the output is the evaluation result of their emotional state.

[0203] Step 6:

[0204] The device provides measures to encourage relaxation in the user based on the emotional state assessment results sent from the server. Specifically, it displays relaxation notifications on the device and automatically sends alerts to emergency contacts as needed. The input is the emotional state assessment results, and the output is relaxation notifications to the user and alerts to external parties.

[0205] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0206] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0207] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0208] [Second Embodiment]

[0209] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0210] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0211] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0212] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0213] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0215] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0216] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0217] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0218] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0219] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0220] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0221] The crime prevention support system according to the present invention is a technology that uses visitor voice and video data to perform real-time analysis in order to ensure the safety of the elderly. This system consists of a server, terminals, and users, each performing the following functions.

[0222] The server receives audio and video data from sensors and performs analysis to identify visitors. Specifically, it uses a machine learning model to recognize visitors' faces and compares them with information registered in a database to determine whether the visitor is a pre-authorized individual. If an unregistered visitor is detected during this process, it is identified as a suspicious person, and information is generated to issue a warning.

[0223] The terminal receives warning information sent from the server and issues warnings to the user. The terminal uses push notifications and voice alarms to alert elderly individuals to the presence of suspicious persons at the appropriate time. Furthermore, it has a function that enables remote communication with visitors based on security camera data, allowing elderly individuals to check the visitor's situation in real time.

[0224] Users can receive warnings and notifications via their devices and communicate securely with external visitors. This reduces anxiety in daily life and ensures safety at home. If necessary, users can report the situation to the local crime prevention network, contributing to broader crime prevention efforts.

[0225] For example, if an unfamiliar visitor C appears at an elderly person's home one day, the server immediately analyzes the audio and video data and performs facial recognition. If the result reveals that visitor C is an unregistered person, they are identified as a suspicious person, and a warning is sent to the elderly person via their device. Upon receiving this information, the elderly person can use the device's camera system to interact with visitor C and confirm the situation.

[0226] This allows elderly people to live their daily lives with peace of mind and enables prompt responses as needed. This system will make a significant contribution to ensuring safety in today's aging society.

[0227] The following describes the processing flow.

[0228] Step 1:

[0229] The server receives audio and video data from the sensors in real time. This data is used as input for analysis necessary to identify visitors.

[0230] Step 2:

[0231] The server uses the received video data to apply a machine learning model and recognize the visitor's face. This process extracts facial features from the video and compares them with an existing database.

[0232] Step 3:

[0233] The server checks the analysis results and determines whether the identified face is registered in the database. If the face is not registered, it is identified as a suspicious person and suspicious person information is generated.

[0234] Step 4:

[0235] The server analyzes the audio data and detects unusual sounds (e.g., loud noises or shouting). Based on these analysis results, it sets triggers for issuing warnings.

[0236] Step 5:

[0237] The device receives warning information sent from the server and sends a notification to the user. This notification is provided as a push notification or an audio alarm.

[0238] Step 6:

[0239] The device activates a communication mechanism for remotely interacting with visitors. This allows the user to check the visitor's status via the camera.

[0240] Step 7:

[0241] Based on the warnings received through the device, the user checks the visitor's status and determines the necessary action. Only after confirming the visitor's safety will the door be unlocked.

[0242] Step 8:

[0243] The server shares surrounding information with the local crime prevention network and takes the initiative as part of crime prevention measures for the entire region.

[0244] Step 9:

[0245] Users can improve crime prevention by reporting warning information to their local community as needed and strengthening cooperation.

[0246] (Example 1)

[0247] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0248] In modern society, ensuring the safety of elderly people in their homes is a critical issue. It is necessary to quickly and accurately identify visitors and prevent intruders from entering homes. However, conventional systems struggle with real-time responses, posing challenges to information sharing within the community and the rapid issuance of warnings.

[0249] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0250] In this invention, the server includes processing means for receiving and processing audio and video data in real time, identification means for identifying visitors identified by the processing means as unidentified individuals if they are not pre-registered, and notification means for sending a warning based on the result of identifying an individual as unidentified. This enables elderly people to quickly receive information about suspicious individuals and to implement effective crime prevention measures in cooperation with local networks.

[0251] "Audio data" refers to digitized data that includes visitors' voice information and is used for visitor identification and behavioral analysis.

[0252] "Video data" refers to digital video containing the visitor's visual information, and is image data used for facial recognition and motion analysis.

[0253] "Processing means" refers to computing devices and programs used to analyze acoustic and video data to identify visitors and analyze their behavior.

[0254] A "discrimination means" is a device that has the function of determining whether a visitor is a person who has been registered in advance, based on the analyzed acoustic and video data.

[0255] "Notification means" refers to devices or systems that promptly transmit information about identified unidentified individuals as a warning to elderly people and relevant organizations.

[0256] "Communication means" refers to communication technologies or equipment used to enable remote interaction with visitors.

[0257] "Information sharing means" refers to networks and technologies for sharing information about visitors with other stakeholders and organizations in order to improve safety within a region.

[0258] This invention provides a security system that receives and analyzes visitors' audio and video data in real time to ensure the safety of the elderly.

[0259] Specifically, the server receives audio and video data from audio and video sensors installed at entrances and other locations. The received data is analyzed using image processing libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow). Through this analysis, visitor facial recognition is performed, and the visitor's permission status is determined by comparing it with pre-registered information stored in a database. If an unregistered visitor is detected, the server identifies them as a suspicious person, generates warning information, and sends it to the terminal.

[0260] The terminal is responsible for transmitting warning information sent from the server to the elderly. The terminal consists of smartphones and tablets, and promptly displays warnings via push notification services (e.g., Firebase Cloud Messaging). Furthermore, the terminal allows the user to interact remotely with visitors via its built-in camera and microphone. This interaction function enables the elderly to check the visitor's situation and take appropriate action as needed.

[0261] Users make decisions to ensure the safety of their homes based on warnings and notifications received via their devices. Users can verify the faces and voices of visitors or report the situation to the local security network using an application on their device. This process strengthens coordinated crime prevention measures across the entire community.

[0262] For example, if a suspicious person approaches the front door of an elderly person's home, the server immediately analyzes the data and sends a warning to the terminal. The user can view the video of the suspicious person displayed on the terminal screen and confirm the situation using the remote dialogue function. In this way, real-time vigilance and information sharing within the community are achieved.

[0263] An example of a prompt message would be, "Please tell me about the specific operation and dialogue functions of the security system developed for the elderly."

[0264] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0265] Step 1:

[0266] The server receives audio and video data in real time from acoustic and video sensors. Inputs include visitor audio and video. This data is securely transmitted to the server using security protocols. Outputs include the received raw data, ready for use in the next analysis step.

[0267] Step 2:

[0268] The server analyzes the received audio and video data. During this process, it uses a facial recognition algorithm to detect visitors' faces and compares them to facial images registered in a database. The input is raw audio and video data, and the output is specific facial features and audio information for person identification. Specifically, a Python script uses the OpenCV library to perform face detection.

[0269] Step 3:

[0270] The server determines whether a visitor is a pre-registered or unregistered person based on the analysis results. The input is the visitor's feature data obtained in the previous step, and the output is the determination of whether or not the person is authorized. In this process, a machine learning model is used to match the feature vectors.

[0271] Step 4:

[0272] If the server identifies a person as suspicious based on the identification result, it generates a warning message and sends it to the terminal. In this case, the input is the identification result, and the output is a warning message containing information about the suspicious person. Specifically, the message is generated in JSON format and sent via the network.

[0273] Step 5:

[0274] The terminal receives warning messages from the server and notifies the user of the warning. The input is the warning message sent from the server, and the output is the warning information displayed on the terminal screen and an audio alarm. Specifically, the terminal's notification function is used to generate push notifications and audio alerts to immediately deliver warnings to elderly users.

[0275] Step 6:

[0276] The user interacts remotely with visitors via their device. The input is real-time data from the camera and microphone, while the output is video and audio calls displayed on the device screen. Specifically, the user communicates with visitors through a video chat application, checks the visitor's status, and makes decisions to ensure their safety.

[0277] (Application Example 1)

[0278] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0279] To ensure the safety of the elderly and residents, it is necessary to effectively monitor visitors coming and going and to take immediate action if unregistered individuals or suspicious behavior are detected. However, current security systems struggle to identify individuals and analyze abnormal behavior in real time, and they do not adequately provide immediate notification and countermeasures to facility staff and residents. Furthermore, the lack of secure means of communication with visitors can lead to delays in confirming their situation and responding to such situations.

[0280] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0281] In this invention, the server includes an analysis means for receiving the voice data and video data of visitors in real time, analyzing them to identify people; an identification means for identifying a person as a suspicious person when the person identified by the analysis means is not registered in advance; and a warning transmission means for transmitting a warning based on the result of identification as a suspicious person. Thereby, the monitoring of visitors in the facility is strengthened, and it becomes possible to provide quick and appropriate warnings and information to staff and residents.

[0282] The "analysis means" is a technology for receiving the voice data and video data of visitors in real time and performing person identification therefrom.

[0283] The "identification means" is a method for identifying a person as a suspicious person when the person identified by the analysis means is not registered in advance.

[0284] The "warning transmission means" is a method for transmitting a warning to users and facility staff based on the result of identification as a suspicious person.

[0285] The "means for transmitting information" is a method for transmitting information necessary to ensure the safety of residents to relevant parties in the surrounding environment.

[0286] The "communication means" is a technology for enabling remote conversation with visitors by image and voice and realizing secure communication.

[0287] The "behavior analysis means" is a processing method for analyzing the behavior patterns of visitors based on voice data and video data and detecting abnormal behaviors.

[0288] The "means for transmitting push notifications" is a method for evaluating the safety of visitors in the facility based on the analyzed data and immediately notifying staff and relevant parties as needed.

[0289] The system that realizes this invention mainly consists of a server, a terminal, and a user. The server is responsible for collecting visitor audio and video data in real time. Specifically, it acquires data through the camera and microphone of a smartphone or a fixed device. For data analysis, machine learning frameworks such as TensorFlow Lite and facial recognition technology using OpenCV are used. WebRTC and the like are used for audio analysis.

[0290] The server uses analysis tools to identify visitors based on the acquired data. If the identified visitor is not previously registered in the database, the identification tool recognizes them as a suspicious person, and an immediate warning system is activated. This sends a warning to the terminal. The terminal is a smartphone or mobile device, and the system sends push notifications or voice alerts to inform the user of the emergency.

[0291] This system also includes communication methods that enable secure remote interaction with visitors. Users can interact with visitors in real time via video and audio through their devices. This feature allows users to check on visitors' situations and intervene as needed.

[0292] As a concrete example, staff working at a nursing home use smartphones with the "Safe Visitor Check" app installed, and receive an immediate alert if an unregistered visitor appears. The staff can then interact with the visitor through the device to confirm their safety. An example of a prompt message to the generated AI model in this system is, "Please compare the new visitor's face with the database. If it cannot be recognized, please issue an alert."

[0293] This invention can significantly improve safety within the facility and ensure the peace of mind and security of residents.

[0294] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0295] Step 1:

[0296] The server receives audio and video data from visitors in real time via smartphones and cameras. Input is video and audio obtained through the camera and microphone, and output is data transferred to the server. To receive this data, the device needs to be connected to a Wi-Fi or mobile network.

[0297] Step 2:

[0298] The server performs face recognition using the received video data. The input is the video data acquired in step 1, and the data is processed using OpenCV. The output is facial feature information extracted from the video data. The facial features are identified by a machine learning algorithm and compared with a registered database.

[0299] Step 3:

[0300] The server analyzes the audio data to identify the visitor's voiceprint. The input is the audio data obtained in step 1, and the output is the estimated voiceprint. Using WebRTC-based audio processing technology, the server analyzes the audio data and evaluates voice matching by comparing it with registered voices.

[0301] Step 4:

[0302] The server determines whether the visitor is registered or not based on the analysis results. The input is the output results from steps 2 and 3. The output is a flag indicating whether the visitor is registered or not. If the visitor is not registered, they are identified as a suspicious person.

[0303] Step 5:

[0304] The terminal receives notifications from the server and issues warnings. The input is a suspicious person identification notification from step 4, and the output is a push notification and voice alert to the elderly or staff. Notifications are primarily made using the terminal's notification system.

[0305] Step 6:

[0306] The user conducts real-time image and voice interaction with the visitor via the terminal. The input is the real-time video and voice of the visitor, and the output is the information obtained through communication with the visitor. By opening a secure communication channel through the device, the user can check the situation.

[0307] Step 7:

[0308] The server analyzes the visitor's behavior pattern and detects abnormalities. The input is the continuous voice and video data from Step 1, and the output is a notification indicating the normality or abnormality of the behavior. An AI model generated for behavior analysis is used, and when an abnormality is detected, a prompt text example such as "Please analyze the behavior pattern and detect abnormalities" is sent.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0310] The crime prevention support system according to the present invention provides an advanced technology that not only analyzes the visitor's voice data and video data in real time but also monitors the user's emotional state by combining an emotion engine. This system is composed of a server, a terminal, and a user, and each plays the following roles.

[0311] The server receives the voice data and video data provided by the sensor and performs face recognition and behavior analysis of the visitor. The server also analyzes the user's emotion using the emotion engine and collects data related to the emotional state. Through this analysis, it is judged whether the user is feeling stress or anxiety, and preparations are made to issue warnings according to the situation.

[0312] The device receives warning information sent from the server and provides notifications to the user. This process includes push notifications and voice alarms. It also has a feature that automatically sends an external alert if the emotion engine determines the user is in an emergency stress state. In this way, the user can quickly receive the necessary support and intervention. Furthermore, it includes communication capabilities that enable remote interaction with visitors, providing an environment where the user can see visitors in real time.

[0313] Users respond to visitors based on notifications from their device. Furthermore, their emotional state is monitored through an emotion engine built into the device, improving psychological safety when they are alone. For example, if visitor D appears at the front door, the server immediately analyzes the visitor's data, evaluating facial recognition and behavioral patterns. Simultaneously, if the emotion engine detects signs of stress from the user's voice, the device sends a notification to the user encouraging relaxation and, if necessary, sends an alert to the local network or emergency contacts.

[0314] These features allow elderly people to maintain a more secure living environment. This system provides comprehensive crime prevention measures that take emotional states into consideration, contributing particularly to improved safety and security for the elderly.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The server receives audio and video data from visitors in real time from sensors. Audio data is obtained from microphones, and video data from surveillance cameras.

[0318] Step 2:

[0319] The server analyzes the received video data and uses a machine learning model to recognize the visitor's face. Features extracted from the video are compared with existing data in the database to determine if they are already registered.

[0320] Step 3:

[0321] If an unregistered person is identified, the server identifies that person as a suspicious individual and generates suspicious person information. This information is transmitted to the terminal via a warning system.

[0322] Step 4:

[0323] The server then analyzes the visitor's voice data. This voice analysis detects unusual sounds and unusual voice tones, and the results are used for behavioral analysis.

[0324] Step 5:

[0325] The server uses an emotion engine to analyze the user's voice data and evaluate the user's emotional state. In this process, it analyzes changes in voice patterns and tone to determine the degree of stress and anxiety.

[0326] Step 6:

[0327] The device receives warnings from the server and user sentiment rating information, and sends appropriate notifications to the user. These notifications may include warnings about the presence of suspicious individuals or the user's stress level.

[0328] Step 7:

[0329] Users check warning notifications from their devices and take appropriate action based on the situation. They can also view real-time video footage of visitors and, if safety is confirmed, respond using the remote interaction function.

[0330] Step 8:

[0331] The device automatically sends an alert to an external emergency contact or local security network if it determines that the user is in an emergency stress state.

[0332] Step 9:

[0333] Users can share crime prevention information with their local community and family in conjunction with the alerts they receive, promoting collaborative crime prevention measures. This collaboration improves the overall safety of the community.

[0334] (Example 2)

[0335] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0336] In modern society, crime prevention is a critical issue, and there is a particular need to strengthen security measures at home. However, there is a lack of systems that provide comprehensive measures that take into account visitor identification and the emotional state of users, making it difficult to quickly and accurately identify suspicious individuals and take appropriate action. To solve this problem, it is necessary to analyze visitor data in real time and monitor the psychological state of users.

[0337] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0338] In this invention, the server includes data acquisition means for receiving audio and video data of visitors in real time; analysis means for analyzing the data collected by the data acquisition means to identify individuals; identification means for identifying individuals not previously registered as suspicious based on the results of the person identification; emotion analysis means for analyzing the user's emotional state using the analysis means and evaluating stress and anxiety; warning issuing means for issuing warnings according to the results of identification as a suspicious person and the user's emotional state; information sharing means for sharing surrounding information via a local network; and notification means for providing notifications to the user in a visible manner. This enables a high level of crime prevention effectiveness by integrating visitor identification and user emotion monitoring.

[0339] "Data acquisition means" refers to the components of devices and systems for receiving visitors' audio and video data in real time.

[0340] "Analysis means" refers to a function that processes audio and video data collected by data acquisition means to identify visitors.

[0341] "Identification means" refers to a function that identifies unregistered individuals as suspicious persons based on personal information identified by analysis means.

[0342] "Emotional analysis means" refers to a function that analyzes the user's voice data and evaluates the emotional state contained within it.

[0343] The "warning notification system" is a function that generates and sends a warning based on the identification results of a suspicious person and the user's emotional state.

[0344] "Information sharing means" refers to functions for sharing relevant information with other systems and users via a regional network.

[0345] A "notification method" is a function that delivers visual or auditory warnings or information to the user.

[0346] The security support system of this invention relies on cooperation between a server, a terminal, and a user. First, the server uses highly sensitive sensors and cameras to acquire visitor audio and video data in real time. Specifically, the software used can include general image analysis programs or open-source image processing libraries for face recognition, and a speech recognition engine for audio analysis. For example, OpenCV can be used for image analysis, and the speech recognition engine's API can be used for audio analysis.

[0347] The server analyzes the acquired data to identify visitors and matches them against an existing database using facial recognition technology. The server also analyzes visitor behavior patterns to detect unusual behavior. Machine learning algorithms can be used for this analysis. Furthermore, the server performs sentiment analysis based on the user's voice data to evaluate the user's emotional state.

[0348] The terminal is responsible for notifying the user of information and warnings sent from the server. It uses devices such as smartphones, tablets, and smart speakers as notification methods. The terminal also has a function that, based on the results of emotion analysis, displays a message encouraging relaxation on the screen if it determines that the user is experiencing stress.

[0349] Users are expected to take action in response to notifications from their devices. For example, if a visitor is identified as suspicious, the user can check the alert on their device and take appropriate action immediately. They can also use remote interaction features to communicate with visitors through their device as needed.

[0350] For example, if the system analyzes that "Visitor D is exhibiting suspicious behavior," the terminal will notify the user with the message, "This person may be suspicious. Please ensure your safety." Based on this information, the user can take appropriate action in response to the alert. Furthermore, by inputting a prompt such as, "Please tell me how to improve the user's emotional state," into the generating AI model, the system can learn more efficient countermeasures.

[0351] This allows the system to provide comprehensive security measures that take into account not only the behavior of visitors but also the psychological state of the users. It is particularly beneficial in improving the safety and security of the elderly.

[0352] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0353] Step 1:

[0354] The server acquires audio and video data of visitors in real time from sensors and cameras. This input data includes the visitor's voice characteristics and facial movements. After receiving this data, the server converts it into an appropriate format and prepares it for analysis. Specifically, it digitally processes the audio data to extract features and divides the video data into frames.

[0355] Step 2:

[0356] The server analyzes the acquired audio and video data. For audio data, a speech recognition engine is used to convert speech to text and extract the content of the visitor's speech. For video data, image analysis software is used to recognize the visitor's face and compare it with the registered database. Audio and video are given as input, and the output is the result of person identification. For example, it can determine whether the visitor's face is already registered.

[0357] Step 3:

[0358] The server analyzes visitor behavior patterns based on the analyzed data. It uses machine learning models to perform data calculations that detect patterns that differ from normal behavior. If the analysis results match the patterns of a suspicious person, the server uses identification methods to confirm this. Behavioral data is provided as input, and the output is a result of the behavioral anomaly detection.

[0359] Step 4:

[0360] The server analyzes the user's voice data to assess their emotional state. This is done by extracting and classifying emotional expressions in the voice using emotion analysis software. Based on this analysis, the server evaluates whether the user is experiencing stress or anxiety, and identifies potential alarm triggers. The input is the user's voice data, and the output is the result of the emotional state evaluation.

[0361] Step 5:

[0362] The device receives warning information sent from the server and notifies the user. This notification may include push notifications or audio alarms on a smartphone. The input is the warning information packet data, and the output is implemented as a visual or auditory notification to the user. The user then decides on an action regarding the visitor based on this notification.

[0363] Step 6:

[0364] The user interacts with visitors based on notifications provided by the device. If necessary, they use the device's communication methods to directly communicate with visitors and verify their safety. Once this operation is complete, the system-wide log is updated, and data is accumulated for later analysis. The input is the notification from the device, and the output is the specific action taken by the user.

[0365] (Application Example 2)

[0366] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0367] In modern society, ensuring safety during interactions with visitors and maintaining users' psychological well-being are crucial issues. This requires not only identifying suspicious individuals and analyzing visitor behavior patterns, but also systems that provide real-time dialogue with visitors and offer psychological support by analyzing users' emotional states. For elderly users and those living alone, enhancing psychological safety and reducing anxiety and stress is particularly important, making the development of comprehensive security and psychological support systems a critical challenge.

[0368] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0369] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if they are not registered in advance; and a psychological analysis means that analyzes the user's emotional state from the audio and video data to detect a specific psychological state. This enables a rapid response to suspicious individuals, as well as the provision of notifications and support appropriate to the user's emotional state, thereby improving safety and a sense of security.

[0370] "Analysis means" refers to a device or program that receives visitor audio and video data in real time and processes them to identify the person.

[0371] "Identification means" refers to a device or program for recognizing a person identified by analysis means as a suspicious person when that person has not been registered in advance.

[0372] A "warning signaling device" is a device or program that issues an alarm and notifies the surrounding area based on the result of identifying a person as suspicious.

[0373] An "information sharing means" is a device or program for sharing acquired information with the surrounding area via a regional network.

[0374] "Psychological analysis means" refers to a device or program for analyzing a user's emotional state from audio and video data and detecting a specific psychological state.

[0375] A "measure transmission means" is a device or program for transmitting a corresponding measure when a specific psychological state is detected.

[0376] The system implementing this invention consists of a server, a terminal, and a user. The server receives audio and video data of visitors collected in real time via sensor devices such as cameras and microphones, and uses a face recognition API and an audio analysis API as analysis means to identify individuals and perform behavioral analysis. Furthermore, by utilizing a psychological analysis means using an emotion engine to evaluate the user's emotional state, it is possible to detect specific psychological states.

[0377] The device receives warning information transmitted from the server and provides warnings to the user via push notifications. Furthermore, if psychological analysis determines that the user is experiencing a specific psychological state, such as stress, it sends a notification to encourage relaxation. Additionally, it can automatically send alerts to emergency contacts designated by the user, if necessary.

[0378] Users can respond appropriately to visitors based on notifications provided through their devices. Furthermore, the device's built-in psychological analysis capabilities monitor the user's emotional state, enhancing their psychological safety when alone. For example, if a visitor appears at the front door, the server immediately analyzes the visitor's data and, if necessary, sends a notification to the user encouraging them to relax.

[0379] A concrete example is a scenario where, while a user is alone at home, the system monitors visitors and provides the user with a sense of security. By utilizing a generative AI model, the system can provide the user with information through prompt messages such as, "A sentiment-analyzing security app connected to your home security system will identify visitors and send real-time notifications based on their behavior."

[0380] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0381] Step 1:

[0382] The server receives visitor audio and video data in real time via the camera and microphone. This supplies the server with audio and video streams as data input. The data is then sent to the Face Recognition API and Audio Analysis API to prepare it for image and audio analysis.

[0383] Step 2:

[0384] The server analyzes the input audio and video data to identify visitors and analyze their behavior patterns. It uses a face recognition API to identify individuals and an audio analysis API to analyze the tone and content of visitors' voices. If the analysis identifies suspicious behavior or unregistered visitors, this information is output as identification data.

[0385] Step 3:

[0386] Based on the identification information output by the server, it detects an anomaly and sends a warning to the terminal as a means of issuing a warning. Specifically, it generates and sends a warning message to the user's terminal via a push notification platform (such as Firebase). The input is the identification information, and the output is a notification on the terminal.

[0387] Step 4:

[0388] The device receives a warning from the server and displays a notification to the user. The user is notified of the anomaly in real time via push notifications or audio alarms. The input is a warning message from the server, and the output is a visual and auditory notification to the user.

[0389] Step 5:

[0390] The server uses an emotion engine to analyze the user's voice data and evaluate their emotional state. If a specific psychological state (e.g., stress) is detected by the psychological analysis tool, the evaluation result is output. The input is the user's voice data, and the output is the evaluation result of their emotional state.

[0391] Step 6:

[0392] The device provides measures to encourage relaxation in the user based on the emotional state assessment results sent from the server. Specifically, it displays relaxation notifications on the device and automatically sends alerts to emergency contacts as needed. The input is the emotional state assessment results, and the output is relaxation notifications to the user and alerts to external parties.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0396] [Third Embodiment]

[0397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0409] The crime prevention support system according to the present invention is a technology that uses visitor voice and video data to perform real-time analysis in order to ensure the safety of the elderly. This system consists of a server, terminals, and users, each performing the following functions.

[0410] The server receives audio and video data from sensors and performs analysis to identify visitors. Specifically, it uses a machine learning model to recognize visitors' faces and compares them with information registered in a database to determine whether the visitor is a pre-authorized individual. If an unregistered visitor is detected during this process, it is identified as a suspicious person, and information is generated to issue a warning.

[0411] The terminal receives warning information sent from the server and issues warnings to the user. The terminal uses push notifications and voice alarms to alert elderly individuals to the presence of suspicious persons at the appropriate time. Furthermore, it has a function that enables remote communication with visitors based on security camera data, allowing elderly individuals to check the visitor's situation in real time.

[0412] Users can receive warnings and notifications via their devices and communicate securely with external visitors. This reduces anxiety in daily life and ensures safety at home. If necessary, users can report the situation to the local crime prevention network, contributing to broader crime prevention efforts.

[0413] For example, if an unfamiliar visitor C appears at an elderly person's home one day, the server immediately analyzes the audio and video data and performs facial recognition. If the result reveals that visitor C is an unregistered person, they are identified as a suspicious person, and a warning is sent to the elderly person via their device. Upon receiving this information, the elderly person can use the device's camera system to interact with visitor C and confirm the situation.

[0414] This allows elderly people to live their daily lives with peace of mind and enables prompt responses as needed. This system will make a significant contribution to ensuring safety in today's aging society.

[0415] The following describes the processing flow.

[0416] Step 1:

[0417] The server receives audio and video data from the sensors in real time. This data is used as input for analysis necessary to identify visitors.

[0418] Step 2:

[0419] The server uses the received video data to apply a machine learning model and recognize the visitor's face. This process extracts facial features from the video and compares them with an existing database.

[0420] Step 3:

[0421] The server checks the analysis results and determines whether the identified face is registered in the database. If the face is not registered, it is identified as a suspicious person and suspicious person information is generated.

[0422] Step 4:

[0423] The server analyzes the audio data and detects unusual sounds (e.g., loud noises or shouting). Based on these analysis results, it sets triggers for issuing warnings.

[0424] Step 5:

[0425] The device receives warning information sent from the server and sends a notification to the user. This notification is provided as a push notification or an audio alarm.

[0426] Step 6:

[0427] The device activates a communication mechanism for remotely interacting with visitors. This allows the user to check the visitor's status via the camera.

[0428] Step 7:

[0429] Based on the warnings received through the device, the user checks the visitor's status and determines the necessary action. Only after confirming the visitor's safety will the door be unlocked.

[0430] Step 8:

[0431] The server shares surrounding information with the local crime prevention network and takes the initiative as part of crime prevention measures for the entire region.

[0432] Step 9:

[0433] Users can improve crime prevention by reporting warning information to their local community as needed and strengthening cooperation.

[0434] (Example 1)

[0435] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0436] In modern society, ensuring the safety of elderly people in their homes is a critical issue. It is necessary to quickly and accurately identify visitors and prevent intruders from entering homes. However, conventional systems struggle with real-time responses, posing challenges to information sharing within the community and the rapid issuance of warnings.

[0437] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0438] In this invention, the server includes processing means for receiving and processing audio and video data in real time, identification means for identifying visitors identified by the processing means as unidentified individuals if they are not pre-registered, and notification means for sending a warning based on the result of identifying an individual as unidentified. This enables elderly people to quickly receive information about suspicious individuals and to implement effective crime prevention measures in cooperation with local networks.

[0439] "Audio data" refers to digitized data that includes visitors' voice information and is used for visitor identification and behavioral analysis.

[0440] "Video data" refers to digital video containing the visitor's visual information, and is image data used for facial recognition and motion analysis.

[0441] "Processing means" refers to computing devices and programs used to analyze acoustic and video data to identify visitors and analyze their behavior.

[0442] A "discrimination means" is a device that has the function of determining whether a visitor is a person who has been registered in advance, based on the analyzed acoustic and video data.

[0443] "Notification means" refers to devices or systems that promptly transmit information about identified unidentified individuals as a warning to elderly people and relevant organizations.

[0444] "Communication means" refers to communication technologies or equipment used to enable remote interaction with visitors.

[0445] "Information sharing means" refers to networks and technologies for sharing information about visitors with other stakeholders and organizations in order to improve safety within a region.

[0446] This invention provides a security system that receives and analyzes visitors' audio and video data in real time to ensure the safety of the elderly.

[0447] Specifically, the server receives audio and video data from audio and video sensors installed at entrances and other locations. The received data is analyzed using image processing libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow). Through this analysis, visitor facial recognition is performed, and the visitor's permission status is determined by comparing it with pre-registered information stored in a database. If an unregistered visitor is detected, the server identifies them as a suspicious person, generates warning information, and sends it to the terminal.

[0448] The terminal is responsible for transmitting warning information sent from the server to the elderly. The terminal consists of smartphones and tablets, and promptly displays warnings via push notification services (e.g., Firebase Cloud Messaging). Furthermore, the terminal allows the user to interact remotely with visitors via its built-in camera and microphone. This interaction function enables the elderly to check the visitor's situation and take appropriate action as needed.

[0449] Users make decisions to ensure the safety of their homes based on warnings and notifications received via their devices. Users can verify the faces and voices of visitors or report the situation to the local security network using an application on their device. This process strengthens coordinated crime prevention measures across the entire community.

[0450] For example, if a suspicious person approaches the front door of an elderly person's home, the server immediately analyzes the data and sends a warning to the terminal. The user can view the video of the suspicious person displayed on the terminal screen and confirm the situation using the remote dialogue function. In this way, real-time vigilance and information sharing within the community are achieved.

[0451] An example of a prompt message would be, "Please tell me about the specific operation and dialogue functions of the security system developed for the elderly."

[0452] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0453] Step 1:

[0454] The server receives audio and video data in real time from acoustic and video sensors. Inputs include visitor audio and video. This data is securely transmitted to the server using security protocols. Outputs include the received raw data, ready for use in the next analysis step.

[0455] Step 2:

[0456] The server analyzes the received audio and video data. During this process, it uses a facial recognition algorithm to detect visitors' faces and compares them to facial images registered in a database. The input is raw audio and video data, and the output is specific facial features and audio information for person identification. Specifically, a Python script uses the OpenCV library to perform face detection.

[0457] Step 3:

[0458] The server determines whether a visitor is a pre-registered or unregistered person based on the analysis results. The input is the visitor's feature data obtained in the previous step, and the output is the determination of whether or not the person is authorized. In this process, a machine learning model is used to match the feature vectors.

[0459] Step 4:

[0460] If the server identifies a person as suspicious based on the identification result, it generates a warning message and sends it to the terminal. In this case, the input is the identification result, and the output is a warning message containing information about the suspicious person. Specifically, the message is generated in JSON format and sent via the network.

[0461] Step 5:

[0462] The terminal receives warning messages from the server and notifies the user of the warning. The input is the warning message sent from the server, and the output is the warning information displayed on the terminal screen and an audio alarm. Specifically, the terminal's notification function is used to generate push notifications and audio alerts to immediately deliver warnings to elderly users.

[0463] Step 6:

[0464] The user interacts remotely with visitors via their device. The input is real-time data from the camera and microphone, while the output is video and audio calls displayed on the device screen. Specifically, the user communicates with visitors through a video chat application, checks the visitor's status, and makes decisions to ensure their safety.

[0465] (Application Example 1)

[0466] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0467] To ensure the safety of the elderly and residents, it is necessary to effectively monitor visitors coming and going and to take immediate action if unregistered individuals or suspicious behavior are detected. However, current security systems struggle to identify individuals and analyze abnormal behavior in real time, and they do not adequately provide immediate notification and countermeasures to facility staff and residents. Furthermore, the lack of secure means of communication with visitors can lead to delays in confirming their situation and responding to such situations.

[0468] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0469] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if the person is not registered in advance; and a warning issuing means that issues a warning based on the result of identifying the person as a suspicious person. This enhances visitor monitoring within the facility and enables the provision of prompt and appropriate warnings and information to staff and residents.

[0470] "Analysis means" refers to technology that receives visitor audio and video data in real time and uses that data to identify individuals.

[0471] "Identification means" refers to a method of identifying a person as a suspicious person when that person has not been previously registered by the analysis means.

[0472] A "warning system" is a method for issuing warnings to users or facility staff based on the results of identifying someone as a suspicious person.

[0473] "Means of disseminating information" refers to methods for transmitting information necessary to ensure the safety of residents in the surrounding environment to relevant parties.

[0474] "Communication methods" refer to technologies that enable remote interaction with visitors using images and audio, thereby ensuring secure communication.

[0475] "Behavioral analysis means" refers to a processing method for analyzing visitor behavior patterns based on audio and video data and detecting abnormal behavior.

[0476] "Methods for sending push notifications" refer to methods for evaluating the safety of visitors within a facility based on analyzed data and immediately notifying staff and relevant parties as needed.

[0477] The system that realizes this invention mainly consists of a server, a terminal, and a user. The server is responsible for collecting visitor audio and video data in real time. Specifically, it acquires data through the camera and microphone of a smartphone or a fixed device. For data analysis, machine learning frameworks such as TensorFlow Lite and facial recognition technology using OpenCV are used. WebRTC and the like are used for audio analysis.

[0478] The server uses analysis tools to identify visitors based on the acquired data. If the identified visitor is not previously registered in the database, the identification tool recognizes them as a suspicious person, and an immediate warning system is activated. This sends a warning to the terminal. The terminal is a smartphone or mobile device, and the system sends push notifications or voice alerts to inform the user of the emergency.

[0479] This system also includes communication methods that enable secure remote interaction with visitors. Users can interact with visitors in real time via video and audio through their devices. This feature allows users to check on visitors' situations and intervene as needed.

[0480] As a concrete example, staff working at a nursing home use smartphones with the "Safe Visitor Check" app installed, and receive an immediate alert if an unregistered visitor appears. The staff can then interact with the visitor through the device to confirm their safety. An example of a prompt message to the generated AI model in this system is, "Please compare the new visitor's face with the database. If it cannot be recognized, please issue an alert."

[0481] This invention can significantly improve safety within the facility and ensure the peace of mind and security of residents.

[0482] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0483] Step 1:

[0484] The server receives audio and video data from visitors in real time via smartphones and cameras. Input is video and audio obtained through the camera and microphone, and output is data transferred to the server. To receive this data, the device needs to be connected to a Wi-Fi or mobile network.

[0485] Step 2:

[0486] The server performs face recognition using the received video data. The input is the video data acquired in step 1, and the data is processed using OpenCV. The output is facial feature information extracted from the video data. The facial features are identified by a machine learning algorithm and compared with a registered database.

[0487] Step 3:

[0488] The server analyzes the audio data to identify the visitor's voiceprint. The input is the audio data obtained in step 1, and the output is the estimated voiceprint. Using WebRTC-based audio processing technology, the server analyzes the audio data and evaluates voice matching by comparing it with registered voices.

[0489] Step 4:

[0490] The server determines whether the visitor is registered or not based on the analysis results. The input is the output results from steps 2 and 3. The output is a flag indicating whether the visitor is registered or not. If the visitor is not registered, they are identified as a suspicious person.

[0491] Step 5:

[0492] The terminal receives notifications from the server and issues warnings. The input is a suspicious person identification notification from step 4, and the output is a push notification and voice alert to the elderly or staff. Notifications are primarily made using the terminal's notification system.

[0493] Step 6:

[0494] Users interact with visitors in real time via image and audio through their devices. Input is real-time video and audio of the visitor, and output is information obtained through communication with the visitor. A secure communication channel is opened through the device, allowing users to monitor the situation.

[0495] Step 7:

[0496] The server analyzes visitor behavior patterns and detects anomalies. Input is continuous audio and video data from step 1, and output is a notification indicating whether the behavior is normal or abnormal. A generative AI model is used for behavior analysis, and in the event of an anomaly, a prompt example message such as "Analyze the behavior pattern and detect anomalies" is sent.

[0497] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0498] The crime prevention support system according to the present invention provides advanced technology that not only analyzes visitor voice and video data in real time but also monitors the user's emotional state by combining it with an emotion engine. This system consists of a server, a terminal, and a user, each playing the following roles.

[0499] The server receives audio and video data from sensors and performs visitor facial recognition and behavioral analysis. The server also uses an emotion engine to analyze the user's emotions and collect data related to their emotional state. This analysis helps determine if the user is experiencing stress or anxiety and prepares to issue situation-appropriate warnings.

[0500] The device receives warning information sent from the server and provides notifications to the user. This process includes push notifications and voice alarms. It also has a feature that automatically sends an external alert if the emotion engine determines the user is in an emergency stress state. In this way, the user can quickly receive the necessary support and intervention. Furthermore, it includes communication capabilities that enable remote interaction with visitors, providing an environment where the user can see visitors in real time.

[0501] Users respond to visitors based on notifications from their device. Furthermore, their emotional state is monitored through an emotion engine built into the device, improving psychological safety when they are alone. For example, if visitor D appears at the front door, the server immediately analyzes the visitor's data, evaluating facial recognition and behavioral patterns. Simultaneously, if the emotion engine detects signs of stress from the user's voice, the device sends a notification to the user encouraging relaxation and, if necessary, sends an alert to the local network or emergency contacts.

[0502] These features allow elderly people to maintain a more secure living environment. This system provides comprehensive crime prevention measures that take emotional states into consideration, contributing particularly to improved safety and security for the elderly.

[0503] The following describes the processing flow.

[0504] Step 1:

[0505] The server receives audio and video data from visitors in real time from sensors. Audio data is obtained from microphones, and video data from surveillance cameras.

[0506] Step 2:

[0507] The server analyzes the received video data and uses a machine learning model to recognize the visitor's face. Features extracted from the video are compared with existing data in the database to determine if they are already registered.

[0508] Step 3:

[0509] If an unregistered person is identified, the server identifies that person as a suspicious individual and generates suspicious person information. This information is transmitted to the terminal via a warning system.

[0510] Step 4:

[0511] The server then analyzes the visitor's voice data. This voice analysis detects unusual sounds and unusual voice tones, and the results are used for behavioral analysis.

[0512] Step 5:

[0513] The server uses an emotion engine to analyze the user's voice data and evaluate the user's emotional state. In this process, it analyzes changes in voice patterns and tone to determine the degree of stress and anxiety.

[0514] Step 6:

[0515] The device receives warnings from the server and user sentiment rating information, and sends appropriate notifications to the user. These notifications may include warnings about the presence of suspicious individuals or the user's stress level.

[0516] Step 7:

[0517] Users check warning notifications from their devices and take appropriate action based on the situation. They can also view real-time video footage of visitors and, if safety is confirmed, respond using the remote interaction function.

[0518] Step 8:

[0519] The device automatically sends an alert to an external emergency contact or local security network if it determines that the user is in an emergency stress state.

[0520] Step 9:

[0521] Users can share crime prevention information with their local community and family in conjunction with the alerts they receive, promoting collaborative crime prevention measures. This collaboration improves the overall safety of the community.

[0522] (Example 2)

[0523] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] In modern society, crime prevention is a critical issue, and there is a particular need to strengthen security measures at home. However, there is a lack of systems that provide comprehensive measures that take into account visitor identification and the emotional state of users, making it difficult to quickly and accurately identify suspicious individuals and take appropriate action. To solve this problem, it is necessary to analyze visitor data in real time and monitor the psychological state of users.

[0525] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0526] In this invention, the server includes data acquisition means for receiving audio and video data of visitors in real time; analysis means for analyzing the data collected by the data acquisition means to identify individuals; identification means for identifying individuals not previously registered as suspicious based on the results of the person identification; emotion analysis means for analyzing the user's emotional state using the analysis means and evaluating stress and anxiety; warning issuing means for issuing warnings according to the results of identification as a suspicious person and the user's emotional state; information sharing means for sharing surrounding information via a local network; and notification means for providing notifications to the user in a visible manner. This enables a high level of crime prevention effectiveness by integrating visitor identification and user emotion monitoring.

[0527] "Data acquisition means" refers to the components of devices and systems for receiving visitors' audio and video data in real time.

[0528] "Analysis means" refers to a function that processes audio and video data collected by data acquisition means to identify visitors.

[0529] "Identification means" refers to a function that identifies unregistered individuals as suspicious persons based on personal information identified by analysis means.

[0530] "Emotional analysis means" refers to a function that analyzes the user's voice data and evaluates the emotional state contained within it.

[0531] The "warning notification system" is a function that generates and sends a warning based on the identification results of a suspicious person and the user's emotional state.

[0532] "Information sharing means" refers to functions for sharing relevant information with other systems and users via a regional network.

[0533] A "notification method" is a function that delivers visual or auditory warnings or information to the user.

[0534] The security support system of this invention relies on cooperation between a server, a terminal, and a user. First, the server uses highly sensitive sensors and cameras to acquire visitor audio and video data in real time. Specifically, the software used can include general image analysis programs or open-source image processing libraries for face recognition, and a speech recognition engine for audio analysis. For example, OpenCV can be used for image analysis, and the speech recognition engine's API can be used for audio analysis.

[0535] The server analyzes the acquired data to identify visitors and matches them against an existing database using facial recognition technology. The server also analyzes visitor behavior patterns to detect unusual behavior. Machine learning algorithms can be used for this analysis. Furthermore, the server performs sentiment analysis based on the user's voice data to evaluate the user's emotional state.

[0536] The terminal is responsible for notifying the user of information and warnings sent from the server. It uses devices such as smartphones, tablets, and smart speakers as notification methods. The terminal also has a function that, based on the results of emotion analysis, displays a message encouraging relaxation on the screen if it determines that the user is experiencing stress.

[0537] Users are expected to take action in response to notifications from their devices. For example, if a visitor is identified as suspicious, the user can check the alert on their device and take appropriate action immediately. They can also use remote interaction features to communicate with visitors through their device as needed.

[0538] For example, if the system analyzes that "Visitor D is exhibiting suspicious behavior," the terminal will notify the user with the message, "This person may be suspicious. Please ensure your safety." Based on this information, the user can take appropriate action in response to the alert. Furthermore, by inputting a prompt such as, "Please tell me how to improve the user's emotional state," into the generating AI model, the system can learn more efficient countermeasures.

[0539] This allows the system to provide comprehensive security measures that take into account not only the behavior of visitors but also the psychological state of the users. It is particularly beneficial in improving the safety and security of the elderly.

[0540] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0541] Step 1:

[0542] The server acquires audio and video data of visitors in real time from sensors and cameras. This input data includes the visitor's voice characteristics and facial movements. After receiving this data, the server converts it into an appropriate format and prepares it for analysis. Specifically, it digitally processes the audio data to extract features and divides the video data into frames.

[0543] Step 2:

[0544] The server analyzes the acquired audio and video data. For audio data, a speech recognition engine is used to convert speech to text and extract the content of the visitor's speech. For video data, image analysis software is used to recognize the visitor's face and compare it with the registered database. Audio and video are given as input, and the output is the result of person identification. For example, it can determine whether the visitor's face is already registered.

[0545] Step 3:

[0546] The server analyzes visitor behavior patterns based on the analyzed data. It uses machine learning models to perform data calculations that detect patterns that differ from normal behavior. If the analysis results match the patterns of a suspicious person, the server uses identification methods to confirm this. Behavioral data is provided as input, and the output is a result of the behavioral anomaly detection.

[0547] Step 4:

[0548] The server analyzes the user's voice data to assess their emotional state. This is done by extracting and classifying emotional expressions in the voice using emotion analysis software. Based on this analysis, the server evaluates whether the user is experiencing stress or anxiety, and identifies potential alarm triggers. The input is the user's voice data, and the output is the result of the emotional state evaluation.

[0549] Step 5:

[0550] The device receives warning information sent from the server and notifies the user. This notification may include push notifications or audio alarms on a smartphone. The input is the warning information packet data, and the output is implemented as a visual or auditory notification to the user. The user then decides on an action regarding the visitor based on this notification.

[0551] Step 6:

[0552] The user interacts with visitors based on notifications provided by the device. If necessary, they use the device's communication methods to directly communicate with visitors and verify their safety. Once this operation is complete, the system-wide log is updated, and data is accumulated for later analysis. The input is the notification from the device, and the output is the specific action taken by the user.

[0553] (Application Example 2)

[0554] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0555] In modern society, ensuring safety during interactions with visitors and maintaining users' psychological well-being are crucial issues. This requires not only identifying suspicious individuals and analyzing visitor behavior patterns, but also systems that provide real-time dialogue with visitors and offer psychological support by analyzing users' emotional states. For elderly users and those living alone, enhancing psychological safety and reducing anxiety and stress is particularly important, making the development of comprehensive security and psychological support systems a critical challenge.

[0556] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0557] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if they are not registered in advance; and a psychological analysis means that analyzes the user's emotional state from the audio and video data to detect a specific psychological state. This enables a rapid response to suspicious individuals, as well as the provision of notifications and support appropriate to the user's emotional state, thereby improving safety and a sense of security.

[0558] "Analysis means" refers to a device or program that receives visitor audio and video data in real time and processes them to identify the person.

[0559] "Identification means" refers to a device or program for recognizing a person identified by analysis means as a suspicious person when that person has not been registered in advance.

[0560] A "warning signaling device" is a device or program that issues an alarm and notifies the surrounding area based on the result of identifying a person as suspicious.

[0561] An "information sharing means" is a device or program for sharing acquired information with the surrounding area via a regional network.

[0562] "Psychological analysis means" refers to a device or program for analyzing a user's emotional state from audio and video data and detecting a specific psychological state.

[0563] A "measure transmission means" is a device or program for transmitting a corresponding measure when a specific psychological state is detected.

[0564] The system implementing this invention consists of a server, a terminal, and a user. The server receives audio and video data of visitors collected in real time via sensor devices such as cameras and microphones, and uses a face recognition API and an audio analysis API as analysis means to identify individuals and perform behavioral analysis. Furthermore, by utilizing a psychological analysis means using an emotion engine to evaluate the user's emotional state, it is possible to detect specific psychological states.

[0565] The device receives warning information transmitted from the server and provides warnings to the user via push notifications. Furthermore, if psychological analysis determines that the user is experiencing a specific psychological state, such as stress, it sends a notification to encourage relaxation. Additionally, it can automatically send alerts to emergency contacts designated by the user, if necessary.

[0566] Users can respond appropriately to visitors based on notifications provided through their devices. Furthermore, the device's built-in psychological analysis capabilities monitor the user's emotional state, enhancing their psychological safety when alone. For example, if a visitor appears at the front door, the server immediately analyzes the visitor's data and, if necessary, sends a notification to the user encouraging them to relax.

[0567] A concrete example is a scenario where, while a user is alone at home, the system monitors visitors and provides the user with a sense of security. By utilizing a generative AI model, the system can provide the user with information through prompt messages such as, "A sentiment-analyzing security app connected to your home security system will identify visitors and send real-time notifications based on their behavior."

[0568] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0569] Step 1:

[0570] The server receives visitor audio and video data in real time via the camera and microphone. This supplies the server with audio and video streams as data input. The data is then sent to the Face Recognition API and Audio Analysis API to prepare it for image and audio analysis.

[0571] Step 2:

[0572] The server analyzes the input audio and video data to identify visitors and analyze their behavior patterns. It uses a face recognition API to identify individuals and an audio analysis API to analyze the tone and content of visitors' voices. If the analysis identifies suspicious behavior or unregistered visitors, this information is output as identification data.

[0573] Step 3:

[0574] Based on the identification information output by the server, it detects an anomaly and sends a warning to the terminal as a means of issuing a warning. Specifically, it generates and sends a warning message to the user's terminal via a push notification platform (such as Firebase). The input is the identification information, and the output is a notification on the terminal.

[0575] Step 4:

[0576] The device receives a warning from the server and displays a notification to the user. The user is notified of the anomaly in real time via push notifications or audio alarms. The input is a warning message from the server, and the output is a visual and auditory notification to the user.

[0577] Step 5:

[0578] The server uses an emotion engine to analyze the user's voice data and evaluate their emotional state. If a specific psychological state (e.g., stress) is detected by the psychological analysis tool, the evaluation result is output. The input is the user's voice data, and the output is the evaluation result of their emotional state.

[0579] Step 6:

[0580] The device provides measures to encourage relaxation in the user based on the emotional state assessment results sent from the server. Specifically, it displays relaxation notifications on the device and automatically sends alerts to emergency contacts as needed. The input is the emotional state assessment results, and the output is relaxation notifications to the user and alerts to external parties.

[0581] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0582] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0584] [Fourth Embodiment]

[0585] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0586] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0587] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0589] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0591] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0592] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0593] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0594] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0595] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0596] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0597] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0598] The crime prevention support system according to the present invention is a technology that uses visitor voice and video data to perform real-time analysis in order to ensure the safety of the elderly. This system consists of a server, terminals, and users, each performing the following functions.

[0599] The server receives audio and video data from sensors and performs analysis to identify visitors. Specifically, it uses a machine learning model to recognize visitors' faces and compares them with information registered in a database to determine whether the visitor is a pre-authorized individual. If an unregistered visitor is detected during this process, it is identified as a suspicious person, and information is generated to issue a warning.

[0600] The terminal receives warning information sent from the server and issues warnings to the user. The terminal uses push notifications and voice alarms to alert elderly individuals to the presence of suspicious persons at the appropriate time. Furthermore, it has a function that enables remote communication with visitors based on security camera data, allowing elderly individuals to check the visitor's situation in real time.

[0601] Users can receive warnings and notifications via their devices and communicate securely with external visitors. This reduces anxiety in daily life and ensures safety at home. If necessary, users can report the situation to the local crime prevention network, contributing to broader crime prevention efforts.

[0602] For example, if an unfamiliar visitor C appears at an elderly person's home one day, the server immediately analyzes the audio and video data and performs facial recognition. If the result reveals that visitor C is an unregistered person, they are identified as a suspicious person, and a warning is sent to the elderly person via their device. Upon receiving this information, the elderly person can use the device's camera system to interact with visitor C and confirm the situation.

[0603] This allows elderly people to live their daily lives with peace of mind and enables prompt responses as needed. This system will make a significant contribution to ensuring safety in today's aging society.

[0604] The following describes the processing flow.

[0605] Step 1:

[0606] The server receives audio and video data from the sensors in real time. This data is used as input for analysis necessary to identify visitors.

[0607] Step 2:

[0608] The server uses the received video data to apply a machine learning model and recognize the visitor's face. This process extracts facial features from the video and compares them with an existing database.

[0609] Step 3:

[0610] The server checks the analysis results and determines whether the identified face is registered in the database. If the face is not registered, it is identified as a suspicious person and suspicious person information is generated.

[0611] Step 4:

[0612] The server analyzes the audio data and detects unusual sounds (e.g., loud noises or shouting). Based on these analysis results, it sets triggers for issuing warnings.

[0613] Step 5:

[0614] The device receives warning information sent from the server and sends a notification to the user. This notification is provided as a push notification or an audio alarm.

[0615] Step 6:

[0616] The device activates a communication mechanism for remotely interacting with visitors. This allows the user to check the visitor's status via the camera.

[0617] Step 7:

[0618] Based on the warnings received through the device, the user checks the visitor's status and determines the necessary action. Only after confirming the visitor's safety will the door be unlocked.

[0619] Step 8:

[0620] The server shares surrounding information with the local crime prevention network and takes the initiative as part of crime prevention measures for the entire region.

[0621] Step 9:

[0622] Users can improve crime prevention by reporting warning information to their local community as needed and strengthening cooperation.

[0623] (Example 1)

[0624] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0625] In modern society, ensuring the safety of elderly people in their homes is a critical issue. It is necessary to quickly and accurately identify visitors and prevent intruders from entering homes. However, conventional systems struggle with real-time responses, posing challenges to information sharing within the community and the rapid issuance of warnings.

[0626] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0627] In this invention, the server includes processing means for receiving and processing audio and video data in real time, identification means for identifying visitors identified by the processing means as unidentified individuals if they are not pre-registered, and notification means for sending a warning based on the result of identifying an individual as unidentified. This enables elderly people to quickly receive information about suspicious individuals and to implement effective crime prevention measures in cooperation with local networks.

[0628] "Audio data" refers to digitized data that includes visitors' voice information and is used for visitor identification and behavioral analysis.

[0629] "Video data" refers to digital video containing the visitor's visual information, and is image data used for facial recognition and motion analysis.

[0630] "Processing means" refers to computing devices and programs used to analyze acoustic and video data to identify visitors and analyze their behavior.

[0631] A "discrimination means" is a device that has the function of determining whether a visitor is a person who has been registered in advance, based on the analyzed acoustic and video data.

[0632] "Notification means" refers to devices or systems that promptly transmit information about identified unidentified individuals as a warning to elderly people and relevant organizations.

[0633] "Communication means" refers to communication technologies or equipment used to enable remote interaction with visitors.

[0634] "Information sharing means" refers to networks and technologies for sharing information about visitors with other stakeholders and organizations in order to improve safety within a region.

[0635] This invention provides a security system that receives and analyzes visitors' audio and video data in real time to ensure the safety of the elderly.

[0636] Specifically, the server receives audio and video data from audio and video sensors installed at entrances and other locations. The received data is analyzed using image processing libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow). Through this analysis, visitor facial recognition is performed, and the visitor's permission status is determined by comparing it with pre-registered information stored in a database. If an unregistered visitor is detected, the server identifies them as a suspicious person, generates warning information, and sends it to the terminal.

[0637] The terminal is responsible for transmitting warning information sent from the server to the elderly. The terminal consists of smartphones and tablets, and promptly displays warnings via push notification services (e.g., Firebase Cloud Messaging). Furthermore, the terminal allows the user to interact remotely with visitors via its built-in camera and microphone. This interaction function enables the elderly to check the visitor's situation and take appropriate action as needed.

[0638] Users make decisions to ensure the safety of their homes based on warnings and notifications received via their devices. Users can verify the faces and voices of visitors or report the situation to the local security network using an application on their device. This process strengthens coordinated crime prevention measures across the entire community.

[0639] For example, if a suspicious person approaches the front door of an elderly person's home, the server immediately analyzes the data and sends a warning to the terminal. The user can view the video of the suspicious person displayed on the terminal screen and confirm the situation using the remote dialogue function. In this way, real-time vigilance and information sharing within the community are achieved.

[0640] An example of a prompt message would be, "Please tell me about the specific operation and dialogue functions of the security system developed for the elderly."

[0641] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0642] Step 1:

[0643] The server receives audio and video data in real time from acoustic and video sensors. Inputs include visitor audio and video. This data is securely transmitted to the server using security protocols. Outputs include the received raw data, ready for use in the next analysis step.

[0644] Step 2:

[0645] The server analyzes the received audio and video data. During this process, it uses a facial recognition algorithm to detect visitors' faces and compares them to facial images registered in a database. The input is raw audio and video data, and the output is specific facial features and audio information for person identification. Specifically, a Python script uses the OpenCV library to perform face detection.

[0646] Step 3:

[0647] The server determines whether a visitor is a pre-registered or unregistered person based on the analysis results. The input is the visitor's feature data obtained in the previous step, and the output is the determination of whether or not the person is authorized. In this process, a machine learning model is used to match the feature vectors.

[0648] Step 4:

[0649] If the server identifies a person as suspicious based on the identification result, it generates a warning message and sends it to the terminal. In this case, the input is the identification result, and the output is a warning message containing information about the suspicious person. Specifically, the message is generated in JSON format and sent via the network.

[0650] Step 5:

[0651] The terminal receives warning messages from the server and notifies the user of the warning. The input is the warning message sent from the server, and the output is the warning information displayed on the terminal screen and an audio alarm. Specifically, the terminal's notification function is used to generate push notifications and audio alerts to immediately deliver warnings to elderly users.

[0652] Step 6:

[0653] The user interacts remotely with visitors via their device. The input is real-time data from the camera and microphone, while the output is video and audio calls displayed on the device screen. Specifically, the user communicates with visitors through a video chat application, checks the visitor's status, and makes decisions to ensure their safety.

[0654] (Application Example 1)

[0655] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] To ensure the safety of the elderly and residents, it is necessary to effectively monitor visitors coming and going and to take immediate action if unregistered individuals or suspicious behavior are detected. However, current security systems struggle to identify individuals and analyze abnormal behavior in real time, and they do not adequately provide immediate notification and countermeasures to facility staff and residents. Furthermore, the lack of secure means of communication with visitors can lead to delays in confirming their situation and responding to such situations.

[0657] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0658] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if the person is not registered in advance; and a warning issuing means that issues a warning based on the result of identifying the person as a suspicious person. This enhances visitor monitoring within the facility and enables the provision of prompt and appropriate warnings and information to staff and residents.

[0659] "Analysis means" refers to technology that receives visitor audio and video data in real time and uses that data to identify individuals.

[0660] "Identification means" refers to a method of identifying a person as a suspicious person when that person has not been previously registered by the analysis means.

[0661] A "warning system" is a method for issuing warnings to users or facility staff based on the results of identifying someone as a suspicious person.

[0662] "Means of disseminating information" refers to methods for transmitting information necessary to ensure the safety of residents in the surrounding environment to relevant parties.

[0663] "Communication methods" refer to technologies that enable remote interaction with visitors using images and audio, thereby ensuring secure communication.

[0664] "Behavioral analysis means" refers to a processing method for analyzing visitor behavior patterns based on audio and video data and detecting abnormal behavior.

[0665] "Methods for sending push notifications" refer to methods for evaluating the safety of visitors within a facility based on analyzed data and immediately notifying staff and relevant parties as needed.

[0666] The system that realizes this invention mainly consists of a server, a terminal, and a user. The server is responsible for collecting visitor audio and video data in real time. Specifically, it acquires data through the camera and microphone of a smartphone or a fixed device. For data analysis, machine learning frameworks such as TensorFlow Lite and facial recognition technology using OpenCV are used. WebRTC and the like are used for audio analysis.

[0667] The server uses analysis tools to identify visitors based on the acquired data. If the identified visitor is not previously registered in the database, the identification tool recognizes them as a suspicious person, and an immediate warning system is activated. This sends a warning to the terminal. The terminal is a smartphone or mobile device, and the system sends push notifications or voice alerts to inform the user of the emergency.

[0668] This system also includes communication methods that enable secure remote interaction with visitors. Users can interact with visitors in real time via video and audio through their devices. This feature allows users to check on visitors' situations and intervene as needed.

[0669] As a concrete example, staff working at a nursing home use smartphones with the "Safe Visitor Check" app installed, and receive an immediate alert if an unregistered visitor appears. The staff can then interact with the visitor through the device to confirm their safety. An example of a prompt message to the generated AI model in this system is, "Please compare the new visitor's face with the database. If it cannot be recognized, please issue an alert."

[0670] This invention can significantly improve safety within the facility and ensure the peace of mind and security of residents.

[0671] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0672] Step 1:

[0673] The server receives audio and video data from visitors in real time via smartphones and cameras. Input is video and audio obtained through the camera and microphone, and output is data transferred to the server. To receive this data, the device needs to be connected to a Wi-Fi or mobile network.

[0674] Step 2:

[0675] The server performs face recognition using the received video data. The input is the video data acquired in step 1, and the data is processed using OpenCV. The output is facial feature information extracted from the video data. The facial features are identified by a machine learning algorithm and compared with a registered database.

[0676] Step 3:

[0677] The server analyzes the audio data to identify the visitor's voiceprint. The input is the audio data obtained in step 1, and the output is the estimated voiceprint. Using WebRTC-based audio processing technology, the server analyzes the audio data and evaluates voice matching by comparing it with registered voices.

[0678] Step 4:

[0679] The server determines whether the visitor is registered or not based on the analysis results. The input is the output results from steps 2 and 3. The output is a flag indicating whether the visitor is registered or not. If the visitor is not registered, they are identified as a suspicious person.

[0680] Step 5:

[0681] The terminal receives notifications from the server and issues warnings. The input is a suspicious person identification notification from step 4, and the output is a push notification and voice alert to the elderly or staff. Notifications are primarily made using the terminal's notification system.

[0682] Step 6:

[0683] Users interact with visitors in real time via image and audio through their devices. Input is real-time video and audio of the visitor, and output is information obtained through communication with the visitor. A secure communication channel is opened through the device, allowing users to monitor the situation.

[0684] Step 7:

[0685] The server analyzes visitor behavior patterns and detects anomalies. Input is continuous audio and video data from step 1, and output is a notification indicating whether the behavior is normal or abnormal. A generative AI model is used for behavior analysis, and in the event of an anomaly, a prompt example message such as "Analyze the behavior pattern and detect anomalies" is sent.

[0686] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0687] The crime prevention support system according to the present invention provides advanced technology that not only analyzes visitor voice and video data in real time but also monitors the user's emotional state by combining it with an emotion engine. This system consists of a server, a terminal, and a user, each playing the following roles.

[0688] The server receives audio and video data from sensors and performs visitor facial recognition and behavioral analysis. The server also uses an emotion engine to analyze the user's emotions and collect data related to their emotional state. This analysis helps determine if the user is experiencing stress or anxiety and prepares to issue situation-appropriate warnings.

[0689] The device receives warning information sent from the server and provides notifications to the user. This process includes push notifications and voice alarms. It also has a feature that automatically sends an external alert if the emotion engine determines the user is in an emergency stress state. In this way, the user can quickly receive the necessary support and intervention. Furthermore, it includes communication capabilities that enable remote interaction with visitors, providing an environment where the user can see visitors in real time.

[0690] Users respond to visitors based on notifications from their device. Furthermore, their emotional state is monitored through an emotion engine built into the device, improving psychological safety when they are alone. For example, if visitor D appears at the front door, the server immediately analyzes the visitor's data, evaluating facial recognition and behavioral patterns. Simultaneously, if the emotion engine detects signs of stress from the user's voice, the device sends a notification to the user encouraging relaxation and, if necessary, sends an alert to the local network or emergency contacts.

[0691] These features allow elderly people to maintain a more secure living environment. This system provides comprehensive crime prevention measures that take emotional states into consideration, contributing particularly to improved safety and security for the elderly.

[0692] The following describes the processing flow.

[0693] Step 1:

[0694] The server receives audio and video data from visitors in real time from sensors. Audio data is obtained from microphones, and video data from surveillance cameras.

[0695] Step 2:

[0696] The server analyzes the received video data and uses a machine learning model to recognize the visitor's face. Features extracted from the video are compared with existing data in the database to determine if they are already registered.

[0697] Step 3:

[0698] If an unregistered person is identified, the server identifies that person as a suspicious individual and generates suspicious person information. This information is transmitted to the terminal via a warning system.

[0699] Step 4:

[0700] The server then analyzes the visitor's voice data. This voice analysis detects unusual sounds and unusual voice tones, and the results are used for behavioral analysis.

[0701] Step 5:

[0702] The server uses an emotion engine to analyze the user's voice data and evaluate the user's emotional state. In this process, it analyzes changes in voice patterns and tone to determine the degree of stress and anxiety.

[0703] Step 6:

[0704] The device receives warnings from the server and user sentiment rating information, and sends appropriate notifications to the user. These notifications may include warnings about the presence of suspicious individuals or the user's stress level.

[0705] Step 7:

[0706] Users check warning notifications from their devices and take appropriate action based on the situation. They can also view real-time video footage of visitors and, if safety is confirmed, respond using the remote interaction function.

[0707] Step 8:

[0708] The device automatically sends an alert to an external emergency contact or local security network if it determines that the user is in an emergency stress state.

[0709] Step 9:

[0710] Users can share crime prevention information with their local community and family in conjunction with the alerts they receive, promoting collaborative crime prevention measures. This collaboration improves the overall safety of the community.

[0711] (Example 2)

[0712] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] In modern society, crime prevention is a critical issue, and there is a particular need to strengthen security measures at home. However, there is a lack of systems that provide comprehensive measures that take into account visitor identification and the emotional state of users, making it difficult to quickly and accurately identify suspicious individuals and take appropriate action. To solve this problem, it is necessary to analyze visitor data in real time and monitor the psychological state of users.

[0714] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0715] In this invention, the server includes data acquisition means for receiving audio and video data of visitors in real time; analysis means for analyzing the data collected by the data acquisition means to identify individuals; identification means for identifying individuals not previously registered as suspicious based on the results of the person identification; emotion analysis means for analyzing the user's emotional state using the analysis means and evaluating stress and anxiety; warning issuing means for issuing warnings according to the results of identification as a suspicious person and the user's emotional state; information sharing means for sharing surrounding information via a local network; and notification means for providing notifications to the user in a visible manner. This enables a high level of crime prevention effectiveness by integrating visitor identification and user emotion monitoring.

[0716] "Data acquisition means" refers to the components of devices and systems for receiving visitors' audio and video data in real time.

[0717] "Analysis means" refers to a function that processes audio and video data collected by data acquisition means to identify visitors.

[0718] "Identification means" refers to a function that identifies unregistered individuals as suspicious persons based on personal information identified by analysis means.

[0719] "Emotional analysis means" refers to a function that analyzes the user's voice data and evaluates the emotional state contained within it.

[0720] The "warning notification system" is a function that generates and sends a warning based on the identification results of a suspicious person and the user's emotional state.

[0721] "Information sharing means" refers to functions for sharing relevant information with other systems and users via a regional network.

[0722] A "notification method" is a function that delivers visual or auditory warnings or information to the user.

[0723] The security support system of this invention relies on cooperation between a server, a terminal, and a user. First, the server uses highly sensitive sensors and cameras to acquire visitor audio and video data in real time. Specifically, the software used can include general image analysis programs or open-source image processing libraries for face recognition, and a speech recognition engine for audio analysis. For example, OpenCV can be used for image analysis, and the speech recognition engine's API can be used for audio analysis.

[0724] The server analyzes the acquired data to identify visitors and matches them against an existing database using facial recognition technology. The server also analyzes visitor behavior patterns to detect unusual behavior. Machine learning algorithms can be used for this analysis. Furthermore, the server performs sentiment analysis based on the user's voice data to evaluate the user's emotional state.

[0725] The terminal is responsible for notifying the user of information and warnings sent from the server. It uses devices such as smartphones, tablets, and smart speakers as notification methods. The terminal also has a function that, based on the results of emotion analysis, displays a message encouraging relaxation on the screen if it determines that the user is experiencing stress.

[0726] Users are expected to take action in response to notifications from their devices. For example, if a visitor is identified as suspicious, the user can check the alert on their device and take appropriate action immediately. They can also use remote interaction features to communicate with visitors through their device as needed.

[0727] For example, if the system analyzes that "Visitor D is exhibiting suspicious behavior," the terminal will notify the user with the message, "This person may be suspicious. Please ensure your safety." Based on this information, the user can take appropriate action in response to the alert. Furthermore, by inputting a prompt such as, "Please tell me how to improve the user's emotional state," into the generating AI model, the system can learn more efficient countermeasures.

[0728] This allows the system to provide comprehensive security measures that take into account not only the behavior of visitors but also the psychological state of the users. It is particularly beneficial in improving the safety and security of the elderly.

[0729] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0730] Step 1:

[0731] The server acquires audio and video data of visitors in real time from sensors and cameras. This input data includes the visitor's voice characteristics and facial movements. After receiving this data, the server converts it into an appropriate format and prepares it for analysis. Specifically, it digitally processes the audio data to extract features and divides the video data into frames.

[0732] Step 2:

[0733] The server analyzes the acquired audio and video data. For audio data, a speech recognition engine is used to convert speech to text and extract the content of the visitor's speech. For video data, image analysis software is used to recognize the visitor's face and compare it with the registered database. Audio and video are given as input, and the output is the result of person identification. For example, it can determine whether the visitor's face is already registered.

[0734] Step 3:

[0735] The server analyzes visitor behavior patterns based on the analyzed data. It uses machine learning models to perform data calculations that detect patterns that differ from normal behavior. If the analysis results match the patterns of a suspicious person, the server uses identification methods to confirm this. Behavioral data is provided as input, and the output is a result of the behavioral anomaly detection.

[0736] Step 4:

[0737] The server analyzes the user's voice data to assess their emotional state. This is done by extracting and classifying emotional expressions in the voice using emotion analysis software. Based on this analysis, the server evaluates whether the user is experiencing stress or anxiety, and identifies potential alarm triggers. The input is the user's voice data, and the output is the result of the emotional state evaluation.

[0738] Step 5:

[0739] The device receives warning information sent from the server and notifies the user. This notification may include push notifications or audio alarms on a smartphone. The input is the warning information packet data, and the output is implemented as a visual or auditory notification to the user. The user then decides on an action regarding the visitor based on this notification.

[0740] Step 6:

[0741] The user interacts with visitors based on notifications provided by the device. If necessary, they use the device's communication methods to directly communicate with visitors and verify their safety. Once this operation is complete, the system-wide log is updated, and data is accumulated for later analysis. The input is the notification from the device, and the output is the specific action taken by the user.

[0742] (Application Example 2)

[0743] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0744] In modern society, ensuring safety during interactions with visitors and maintaining users' psychological well-being are crucial issues. This requires not only identifying suspicious individuals and analyzing visitor behavior patterns, but also systems that provide real-time dialogue with visitors and offer psychological support by analyzing users' emotional states. For elderly users and those living alone, enhancing psychological safety and reducing anxiety and stress is particularly important, making the development of comprehensive security and psychological support systems a critical challenge.

[0745] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0746] In this invention, the server includes an analysis means that receives audio and video data of visitors in real time and analyzes them to identify the person; an identification means that identifies the person identified by the analysis means as a suspicious person if they are not registered in advance; and a psychological analysis means that analyzes the user's emotional state from the audio and video data to detect a specific psychological state. This enables a rapid response to suspicious individuals, as well as the provision of notifications and support appropriate to the user's emotional state, thereby improving safety and a sense of security.

[0747] "Analysis means" refers to a device or program that receives visitor audio and video data in real time and processes them to identify the person.

[0748] "Identification means" refers to a device or program for recognizing a person identified by analysis means as a suspicious person when that person has not been registered in advance.

[0749] A "warning signaling device" is a device or program that issues an alarm and notifies the surrounding area based on the result of identifying a person as suspicious.

[0750] An "information sharing means" is a device or program for sharing acquired information with the surrounding area via a regional network.

[0751] "Psychological analysis means" refers to a device or program for analyzing a user's emotional state from audio and video data and detecting a specific psychological state.

[0752] A "measure transmission means" is a device or program for transmitting a corresponding measure when a specific psychological state is detected.

[0753] The system implementing this invention consists of a server, a terminal, and a user. The server receives audio and video data of visitors collected in real time via sensor devices such as cameras and microphones, and uses a face recognition API and an audio analysis API as analysis means to identify individuals and perform behavioral analysis. Furthermore, by utilizing a psychological analysis means using an emotion engine to evaluate the user's emotional state, it is possible to detect specific psychological states.

[0754] The device receives warning information transmitted from the server and provides warnings to the user via push notifications. Furthermore, if psychological analysis determines that the user is experiencing a specific psychological state, such as stress, it sends a notification to encourage relaxation. Additionally, it can automatically send alerts to emergency contacts designated by the user, if necessary.

[0755] Users can respond appropriately to visitors based on notifications provided through their devices. Furthermore, the device's built-in psychological analysis capabilities monitor the user's emotional state, enhancing their psychological safety when alone. For example, if a visitor appears at the front door, the server immediately analyzes the visitor's data and, if necessary, sends a notification to the user encouraging them to relax.

[0756] A concrete example is a scenario where, while a user is alone at home, the system monitors visitors and provides the user with a sense of security. By utilizing a generative AI model, the system can provide the user with information through prompt messages such as, "A sentiment-analyzing security app connected to your home security system will identify visitors and send real-time notifications based on their behavior."

[0757] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0758] Step 1:

[0759] The server receives visitor audio and video data in real time via the camera and microphone. This supplies the server with audio and video streams as data input. The data is then sent to the Face Recognition API and Audio Analysis API to prepare it for image and audio analysis.

[0760] Step 2:

[0761] The server analyzes the input audio and video data to identify visitors and analyze their behavior patterns. It uses a face recognition API to identify individuals and an audio analysis API to analyze the tone and content of visitors' voices. If the analysis identifies suspicious behavior or unregistered visitors, this information is output as identification data.

[0762] Step 3:

[0763] Based on the identification information output by the server, it detects an anomaly and sends a warning to the terminal as a means of issuing a warning. Specifically, it generates and sends a warning message to the user's terminal via a push notification platform (such as Firebase). The input is the identification information, and the output is a notification on the terminal.

[0764] Step 4:

[0765] The device receives a warning from the server and displays a notification to the user. The user is notified of the anomaly in real time via push notifications or audio alarms. The input is a warning message from the server, and the output is a visual and auditory notification to the user.

[0766] Step 5:

[0767] The server uses an emotion engine to analyze the user's voice data and evaluate their emotional state. If a specific psychological state (e.g., stress) is detected by the psychological analysis tool, the evaluation result is output. The input is the user's voice data, and the output is the evaluation result of their emotional state.

[0768] Step 6:

[0769] The device provides measures to encourage relaxation in the user based on the emotional state assessment results sent from the server. Specifically, it displays relaxation notifications on the device and automatically sends alerts to emergency contacts as needed. The input is the emotional state assessment results, and the output is relaxation notifications to the user and alerts to external parties.

[0770] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0771] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0773] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0774] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0775] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0776] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0777] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0778] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0779] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0780] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0781] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0782] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0783] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0784] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0785] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0786] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0787] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0788] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0789] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0790] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0791] The following is further disclosed regarding the embodiments described above.

[0792] (Claim 1)

[0793] An analytical means that receives audio and video data of visitors in real time and analyzes them to identify individuals,

[0794] An identification means for identifying a person identified by the aforementioned analysis means as a suspicious person if that person is not registered in advance,

[0795] A warning issuing mechanism that issues a warning based on the result of identifying a person as suspicious,

[0796] Information sharing means for sharing surrounding information via a regional network,

[0797] A system that includes this.

[0798] (Claim 2)

[0799] The system according to claim 1, further comprising a behavioral analysis means for analyzing audio data and video data to analyze visitor behavior patterns and detect abnormal behavior.

[0800] (Claim 3)

[0801] The system according to claim 1, wherein the warning issuing means further includes communication means for remotely interacting with a visitor.

[0802] "Example 1"

[0803] (Claim 1)

[0804] A processing means for receiving and processing audio and video data in real time,

[0805] A determination means for determining a visitor identified by the processing means as an unverified person when the visitor has not been pre-registered,

[0806] A notification system that sends a warning based on the result of identifying an unidentified person,

[0807] Communication means for exchanging information with visitors remotely,

[0808] Information sharing methods that share information about the surrounding area through local connections,

[0809] A system that includes this.

[0810] (Claim 2)

[0811] The system according to claim 1, comprising motion analysis means for analyzing received acoustic and video data, analyzing visitor behavior trends, and detecting anomalies.

[0812] (Claim 3)

[0813] The system according to claim 1, wherein the communication means includes a communication method that enables remote interaction with a visitor.

[0814] "Application Example 1"

[0815] (Claim 1)

[0816] An analytical means that receives audio and video data of visitors in real time and analyzes them to identify individuals,

[0817] An identification means for identifying a person identified by the aforementioned analysis means as a suspicious person if that person is not registered in advance,

[0818] A warning issuing mechanism that issues a warning based on the result of identifying a person as suspicious,

[0819] In the surrounding environment, a means of disseminating information to ensure the safety of residents,

[0820] A communication means that provides remote interaction with visitors using images and audio,

[0821] A system that includes this.

[0822] (Claim 2)

[0823] The system according to claim 1, further comprising a behavioral analysis means for analyzing audio data and video data to analyze visitor behavior patterns and detect abnormal behavior.

[0824] (Claim 3)

[0825] The system according to claim 1, further comprising means for sending push notifications based on analyzed data in order to evaluate the safety of visitors within the facility.

[0826] "Example 2 of combining an emotion engine"

[0827] (Claim 1)

[0828] A data acquisition means for receiving visitor audio and video data in real time,

[0829] An analysis means for identifying a person by analyzing the data collected by the aforementioned data acquisition means,

[0830] An identification method that identifies individuals who are not pre-registered as suspicious based on the results of person identification,

[0831] The aforementioned analysis means analyzes the user's emotional state and evaluates stress and anxiety;

[0832] A warning issuing means that issues a warning based on the result of being identified as a suspicious person and the user's emotional state,

[0833] Information sharing means for sharing surrounding information via a regional network,

[0834] A system that includes a notification mechanism to provide notifications to users in a visible format.

[0835] (Claim 2)

[0836] The system according to claim 1, further comprising a behavior and emotion analysis means for analyzing audio data and video data to analyze visitor behavior patterns, detect abnormal behavior, and evaluate user emotions.

[0837] (Claim 3)

[0838] The system according to claim 1, wherein the warning issuing means further includes communication means for the visitor and the user to communicate in real time.

[0839] "Application example 2 when combining with an emotional engine"

[0840] (Claim 1)

[0841] An analytical means that receives audio and video data of visitors in real time and analyzes them to identify individuals,

[0842] An identification means for identifying a person identified by the aforementioned analysis means as a suspicious person if that person is not registered in advance,

[0843] A warning issuing mechanism that issues a warning based on the result of identifying a person as suspicious,

[0844] Information sharing means for sharing surrounding information via a regional network,

[0845] A psychological analysis means that analyzes the user's emotional state from audio and video data to detect a specific psychological state,

[0846] A means for issuing measures to take appropriate action when a specific psychological state is detected,

[0847] A system that includes this.

[0848] (Claim 2)

[0849] The system according to claim 1, further comprising: behavioral analysis means for analyzing audio and video data to analyze visitor behavior patterns and detect abnormal behavior; and measure transmission means for sending notifications to the user to encourage relaxation when a specific psychological state is detected.

[0850] (Claim 3)

[0851] The system according to claim 1, wherein the warning issuing means further includes a communication means for remotely interacting with a visitor and has a function to automatically send an alert to an external party when a specific psychological state is detected. [Explanation of symbols]

[0852] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An analytical means that receives audio and video data of visitors in real time and analyzes them to identify individuals, An identification means for identifying a person identified by the aforementioned analysis means as a suspicious person if that person is not registered in advance, A warning issuing mechanism that issues a warning based on the result of identifying a person as suspicious, Information sharing means for sharing surrounding information via a regional network, A system that includes this.

2. The system according to claim 1, further comprising a behavioral analysis means for analyzing audio data and video data to analyze visitor behavior patterns and detect abnormal behavior.

3. The system according to claim 1, wherein the warning issuing means further includes communication means for remotely interacting with a visitor.

Images