System

Abstract

A system is provided.SOLUTION: A system including a camera with a human detection sensor, means for receiving data obtained from the camera and detecting presence of a person in a vehicle, means for issuing an alert by voice when a key of the vehicle is removed and a person remains in the vehicle, means for transmitting the alert information to an external communication device, and means for transmitting a notification to a plurality of registered terminals based on the transmitted alert information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, there have been frequent incidents of kindergarten children and young children being left behind in vehicles. As a result, cases of death from heatstroke and other causes have been reported, making this a serious social problem. To prevent such accidents, a system is needed that detects when a person has been left behind in a vehicle and immediately notifies relevant parties. However, current systems have problems such as delayed detection of people and inappropriate notification. The present invention aims to solve these problems and ensure the safety of kindergarten children and young children. [Means for solving the problem]

[0005] This invention first provides a means for detecting the presence of a person inside a vehicle using a camera with a motion sensor. Furthermore, it provides a means for issuing an audio alert if someone remains inside the vehicle when the key is removed. A system is constructed that transmits the alert information to an external communication device and sends notifications to multiple registered terminals, enabling a rapid response.

[0006] In addition, a portable device will be installed to monitor the temperature inside the vehicle, and a means will be installed to issue an audio alert if an abnormal temperature is detected. This alert information will also be sent to an external communication device and notified to multiple terminals, allowing for an immediate response to an emergency.

[0007] Furthermore, when alert information is received, the system includes a means for dialogue using natural language processing technology. This allows the system to gather information about the situation through dialogue with the kindergarten or young child and instruct them on how to seek appropriate help. This dialogue information is also sent to an external communication device, encouraging a quick and appropriate response.

[0008] A "camera with a human sensor" is a device installed inside a vehicle that combines a sensor for detecting the presence of people with a photographic function.

[0009] The "means for detecting the presence of people inside the vehicle" is a system that analyzes data from a camera equipped with a motion sensor to determine whether people remain inside the vehicle.

[0010] "Means for issuing an audio alert if someone remains in the vehicle when the vehicle key is removed" is a device that detects the act of removing the vehicle key and issues an audio alarm if it determines that someone is still in the vehicle.

[0011] "Means for transmitting alert information to an external communication device" refers to a system that notifies an external device (such as a smartphone) of abnormalities detected inside the vehicle (such as the presence of a person or abnormal temperatures) using wireless communication or the Internet.

[0012] A "portable device for monitoring the temperature inside a vehicle" is a portable device that is installed inside a vehicle and continuously monitors the interior temperature using a temperature sensor.

[0013] The "means for detecting abnormal temperatures" is a system that uses a temperature sensor in a portable device to determine that an abnormality has occurred if the temperature exceeds a set temperature threshold.

[0014] "Means for conducting dialogue using natural language processing technology" refers to a system that uses artificial intelligence to conduct dialogue in natural language and collect and analyze information from kindergarteners and young children.

[0015] "Means for gathering information about the situation and instructing how to seek help" is a system that uses information obtained through dialogue to instruct kindergarteners and young children on appropriate behavior and ensure their safety. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention provides a means for preventing accidents in which kindergarten children or small children are left behind in vehicles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[0038] Motion sensor camera operation

[0039] The terminal (camera) is installed inside a vehicle equipped with a motion sensor. This camera periodically collects motion sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the camera checks the latest sensor data at that time and issues an audio alert if someone remains inside. In addition, by sending the alert information to a server, a notification is sent to an external communication device.

[0040] Specific examples

[0041] When the vehicle's engine is shut off and the driver removes the key, the device (camera) detects this action. At the same time, if there are any children remaining in the vehicle, the motion sensor detects their presence and issues an audio alert. In addition, the alert information is sent to the server, and a push notification saying "There are children remaining in the vehicle!" is sent to multiple registered smartphones.

[0042] Operation of a portable device for monitoring temperature inside a vehicle

[0043] The terminal (mobile phone) is used to monitor abnormal temperatures inside the vehicle. This device monitors the temperature inside the vehicle at regular intervals and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server, which then sends a notification to an external communication device.

[0044] Specific examples

[0045] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and determine that the temperature is abnormal if it exceeds 35 degrees. An audio alert will be issued and the alert information will be sent to the server. Based on this information, the server will send a push notification to the registered smartphone saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0046] Dialogue system using natural language processing technology

[0047] The server that receives the alert information activates a dialogue system (such as an AI chatbot) that uses natural language processing technology. This system asks questions to the kindergarten or toddler to confirm the situation. Based on the information obtained through this dialogue, it gives instructions on how to call for help or escape appropriately.

[0048] Specific examples

[0049] When an alert is issued, ChatGPT (registered trademark) will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child responds "Yes," ChatGPT will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," ChatGPT will give instructions such as, "Try to find the door lock button," helping the child escape on their own. The content of this conversation is sent to the server and notified to the registered smartphone as additional information.

[0050] As a result, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and an interactive system using natural language processing technology.

[0051] The processing flow will be explained below.

[0052] Processing for cameras with motion sensors

[0053] Step 1:

[0054] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0055] Step 2:

[0056] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0057] Step 3:

[0058] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[0059] Step 4:

[0060] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[0061] Step 5:

[0062] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[0063] Processing a portable device that monitors the temperature inside a vehicle

[0064] Step 6:

[0065] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[0066] Step 7:

[0067] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[0068] Step 8:

[0069] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[0070] Step 9:

[0071] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[0072] Step 10:

[0073] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[0074] Dialogue processing using natural language processing technology

[0075] Step 11:

[0076] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[0077] Step 12:

[0078] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[0079] Step 13:

[0080] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[0081] Step 14:

[0082] Based on the child's responses, the chatbot will give them instructions on how to escape or call for help, such as "Try to find the door lock button."

[0083] Step 15:

[0084] The chatbot transmits the situation information obtained through the dialogue to the server.

[0085] Step 16:

[0086] The server then sends additional push notifications to the registered smartphone ID based on the collected situation information, such as a notification that "A child is locked in a car. Please check the situation."

[0087] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and a dialogue system using natural language processing technology.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Accidents involving kindergarten children and young children being left behind in vehicles are a serious risk, especially in high-temperature environments. Conventional warning systems only detect the presence of people in the vehicle or abnormally high temperatures, but do not provide specific countermeasures, making immediate rescue difficult in many cases. Furthermore, warnings cannot be properly communicated to nearby adults, resulting in insufficient prevention of accidents. Therefore, there is a need for a system that can protect kindergarten children and young children by providing specific instructions and dialogue tailored to the situation.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for receiving data obtained from an image capture device with a motion sensor and detecting the presence of a person in the vehicle, means for issuing an audio warning if a person remains in the vehicle when the vehicle engine is stopped and the key is removed, means for transmitting the warning information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted warning information, and means for using a generative AI model to inquire about the situation and instruct appropriate responses. This makes it possible to provide not only warnings but also specific escape instructions and calls for help, enabling quick responses to prevent accidents and ensure the safety of kindergarteners and young children.

[0093] An "image capture device with a human presence sensor" is a device that is installed inside a vehicle and combines a sensor and a camera to detect the presence of people.

[0094] "Means for receiving data and detecting the presence of a person inside the vehicle" refers to a device or system that has the function of analyzing the information transmitted from the image capture device and determining whether or not a person is inside the vehicle.

[0095] The "audio warning means" refers to a system that issues an audio alert to people left in the vehicle based on the detection results.

[0096] The "means for transmitting to an external communication device" is a system for transmitting alert information to a mobile phone or an internet-connected device outside the vehicle.

[0097] The "means for sending notifications to multiple registered devices" refers to a device or system that sends alert information in the form of a push notification to specific smartphones or mobile information devices that have been registered in advance.

[0098] "Means of using generative AI models to question the situation and instruct appropriate responses" refers to an AI system that uses natural language processing technology to ask questions of people in the vehicle and instruct them on what to do.

[0099] A "portable information terminal for monitoring the temperature inside a vehicle" is a portable device equipped with a temperature sensor for periodically measuring the temperature inside the vehicle and detecting any abnormalities.

[0100] The "means for detecting abnormal temperatures" is a system that analyzes temperature data and determines whether an abnormality has occurred if the temperature exceeds a set safe range.

[0101] "Means for conducting dialogue using natural language processing technology" refers to a system that uses AI technology to simulate natural conversations with people and grasp the situation.

[0102] "Means for instructing appropriate rescue methods" is a system that provides users with appropriate actions and measures based on collected information.

[0103] The "means for transmitting warning information to an external communication device" is a system for transmitting detected abnormalities or dangerous situations to an external device via a communication network.

[0104] The "means for transmitting situation information to an external communication device" is a system that transfers situation information collected through dialogue to an external device.

[0105] The system of the present invention combines a plurality of devices and technologies to operate systematically in order to prevent accidents in which kindergarten children or small children are left behind in vehicles. Embodiments of the present invention will be described in detail below.

[0106] Motion sensor camera operation

[0107] In this system, a terminal (an image capture device with a human sensor) is installed inside the vehicle. This device is equipped with a sensor and a camera to detect the presence of people inside the vehicle, and periodically collects sensor data. For example, the camera sensor detects whether there is a person inside the vehicle every 10 seconds and stores the detection results in internal memory.

[0108] By monitoring the engine's on / off status, the device detects when the vehicle's engine is turned off and the key is removed. At this time, it checks the latest sensor data and issues an audio warning if anyone remains in the vehicle.

[0109] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends push notifications to multiple pre-registered smartphones and mobile information devices. For example, a warning message such as "There is someone remaining in the vehicle!" is sent.

[0110] Operation of a portable device for monitoring temperature inside a vehicle

[0111] The terminal (personal digital assistant) is used to monitor the temperature inside the vehicle. This terminal is equipped with a temperature sensor and measures the temperature inside the vehicle every minute. If the measured temperature exceeds a set safe range (e.g., 35 degrees), the terminal detects the abnormal temperature and issues an audio warning.

[0112] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends a push notification to multiple pre-registered smartphones and mobile information devices. For example, a message such as "The temperature inside the vehicle is dangerous. Please check immediately!" may be sent.

[0113] Dialogue system using natural language processing technology

[0114] When the server receives the alert information sent from the device, it launches a generative AI model (e.g., ChatGPT) and activates a dialogue system, which asks questions to the kindergartener or toddler to confirm the situation.

[0115] Specifically, the AI ​​chatbot will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," the AI ​​will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," the AI ​​will give instructions such as, "Try to find the door lock button," supporting the child to escape on their own.

[0116] The contents of these conversations are recorded on a server and transferred in real time to an external communication device (smartphone), allowing parents and other relevant parties to immediately understand the situation and take appropriate action.

[0117] Prompt Sentence Examples

[0118] When an alert is triggered, the following prompt is entered into the generative AI model:

[0119] "We have detected that there are children remaining in the vehicle. Please begin a dialogue with the children to confirm their situation."

[0120] This enables the AI ​​model to generate appropriate dialogue depending on the situation, supporting young children and kindergarteners.

[0121] As described above, the system of the present invention can ensure the safety of kindergarten children and young children inside vehicles by combining a camera with a human sensor, a temperature monitoring mobile terminal, and a dialogue system using natural language processing technology.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1:

[0124] The terminal (an image capture device with a motion sensor) collects data from the motion sensor every 10 seconds to detect the presence of people inside the vehicle.

[0125] Specific operation: When the sensor detects the presence of a person, the data (input) is stored in the camera's internal memory. The stored data is output.

[0126] Step 2:

[0127] The terminal monitors the vehicle's engine status and detects when the engine is turned off and the key is removed.

[0128] Specific operation: The system monitors the engine on / off status and key removal timing (input), and makes an alert decision based on that information. The timing of key removal and engine status are output.

[0129] Step 3:

[0130] When the key is removed, the terminal checks the latest motion sensor data to determine whether anyone remains in the vehicle.

[0131] Specific operation: Analyzes the latest stored sensor data (input), and if a person is detected, generates and issues an audio alert. The result of whether or not an alert was issued is output.

[0132] Step 4:

[0133] The terminal transmits the issued alert information to the server.

[0134] Specific operation: Alert information (input) is processed to be sent to an external communication device, and then sent to a server via a network. The server outputs a message indicating that it has received the alert information.

[0135] Step 5:

[0136] The server sends a notification to the registered terminals based on the received alert information.

[0137] Specific operation: The server analyzes the received alert data (input) and sends a push notification to the registered smartphone or information terminal. The result of the notification being sent is output.

[0138] Step 6:

[0139] The terminal (personal digital assistant) periodically monitors the temperature inside the vehicle.

[0140] Specific operation: The temperature sensor measures the temperature inside the vehicle every minute, stores the data (input) in the internal memory, and outputs the measurement results.

[0141] Step 7:

[0142] The terminal analyzes the measurement data and detects an abnormal temperature if the temperature exceeds a set temperature threshold.

[0143] Specific operation: Analyzes the measured in-car temperature data (input) and determines that an abnormality has occurred if the temperature exceeds a preset temperature threshold (e.g., 35°C). The abnormal temperature detection result is output.

[0144] Step 8:

[0145] The device will issue an audio warning if it detects an abnormal temperature.

[0146] Specific operation: Based on the abnormal temperature detection result (input), an audio alert is generated and output.

[0147] Step 9:

[0148] The terminal transmits the warning information to the server.

[0149] Specific operation: The abnormal temperature warning information (input) is processed and sent to the server via the network. The server outputs the information that it has received the warning information.

[0150] Step 10:

[0151] The server sends a notification to multiple registered terminals based on the received abnormal temperature warning information.

[0152] Specific operation: The server analyzes the abnormal temperature data (input) and sends a push notification to registered devices such as smartphones. The notification result is output.

[0153] Step 11:

[0154] The server launches a generative AI model (e.g., ChatGPT) based on the received alert information.

[0155] Specific operation: A prompt sentence (input) saying "There are children remaining in the vehicle. Please check with the children" is sent to the AI ​​model, and a dialogue scenario is generated. The generated dialogue scenario is output.

[0156] Step 12:

[0157] The server then has the generative AI model begin asking questions to the children based on the dialogue scenario.

[0158] Specific operation: Based on the prompt sentence (input), the AI ​​starts a dialogue saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" The progress of the dialogue is output.

[0159] Step 13:

[0160] The server records information obtained through the interaction and sends notifications to external communication devices.

[0161] Specific operation: The situation information (input) collected through the dialogue is stored in a database and transferred to a smartphone in real time. The notification result is output.

[0162] As described above, the various terminals and servers of this system work in cooperation to provide a quick and effective response to protect kindergarten children and young children inside vehicles.

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] Currently, safety management in factories places a heavy burden on workers. It is particularly difficult to grasp and respond to situations when working in high temperatures or approaching dangerous areas, making it necessary to ensure safety quickly and accurately. Furthermore, existing systems do not adequately check the detailed situation when an alert is issued, potentially threatening worker safety. Therefore, a comprehensive system is needed that can efficiently monitor temperatures in factories, detect movement using human sensors, and check the situation and issue instructions in the event of an abnormality.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes: means for receiving data obtained from a motion sensor camera and detecting the presence of people within the area; means for issuing an audio alert if certain conditions are met and a person remains; means for transmitting the alert information to an external communication device; means for sending notifications to multiple registered terminals based on the transmitted alert information; means for conducting a dialogue based on the alert information using natural language processing technology; and means for gathering information on the situation through the dialogue and issuing appropriate instructions. This enables prompt and accurate safety management within the factory. Temperature changes and human activity can be monitored, and specific dialogue can be provided to ensure worker safety in the event of an abnormality.

[0168] A "camera with a human sensor" is a device that detects human movement and collects that information through the camera.

[0169] "Means for detecting the presence of people within an area" refers to technology that uses sensors or cameras to confirm that people are present within a specific area.

[0170] "A means for issuing an audio alert" refers to a device or system that issues an audio warning when an abnormality or emergency is detected.

[0171] "Means for transmitting to an external communication device" refers to technology for transmitting collected data and alert information to other external devices via the Internet or wireless communication.

[0172] The "means for sending notifications to multiple registered terminals" is a system for simultaneously sending notifications to multiple devices, such as smartphones and tablets, that have been registered in advance.

[0173] "Means of conducting dialogue using natural language processing technology" refers to technology that uses AI and machine learning to understand human language and conduct dialogue.

[0174] "Means of gathering information about the situation and issuing appropriate instructions" refers to technology that uses dialogue and sensor data to grasp the situation on-site and issue necessary instructions to people.

[0175] The "means for receiving temperature data and detecting abnormal temperatures" is a technology for receiving data from a temperature sensor and detecting that the data is outside a safe range.

[0176] "Means for generating prompt sentences using a generative AI model" refers to a technology that uses an AI learning model to automatically generate appropriate dialogue sentences or instructions based on specified content.

[0177] System configuration

[0178] This invention is a system that includes a motion sensor camera, a portable temperature monitoring device, a dialogue system using natural language processing technology, and a generative AI model. This system can perform fast and accurate monitoring to ensure the safety of workers in factories.

[0179] Hardware and Software

[0180] Hardware:

[0181] Smart glasses (e.g., Google® Glass®): worn by workers to display alert notifications and interactions.

[0182] Server (e.g., AWS EC2 instance): Collects data, processes data, and sends notifications.

[0183] Temperature sensor (e.g. DHT22): Monitors the temperature inside the factory.

[0184] Camera with motion sensor (e.g. Raspberry Pi Camera Module): Monitors activity within the factory.

[0185] software:

[0186] Natural language processing technology (e.g. ChatGPT): Used to communicate with workers and check the status.

[0187] Generative AI model: Used to generate prompts.

[0188] Data processing and calculation

[0189] The server receives data from cameras equipped with motion sensors and detects the presence of people within the area. Based on this information, if a person meets certain conditions (for example, if they approach a dangerous area), it issues an audio alert. It also continuously receives data from temperature sensors and detects abnormal temperatures when they exceed a set threshold. It then transmits this alert information to an external communications device, which then sends notifications to multiple registered devices.

[0190] Furthermore, a generative AI model is used to generate prompts based on the alert information, and natural language processing technology is used to conduct dialogue, confirming the worker's status in real time and issuing necessary instructions. For example, when a voice alert occurs, the server can use ChatGPT to generate a prompt such as "Please tell us your current situation" and begin dialogue.

[0191] Specific examples

[0192] If a worker working in a factory machine room experiences a sudden temperature rise and reaches 40 degrees, the smart glasses will issue a warning and send a push notification saying, "The temperature in the factory is at a dangerous level. Please evacuate!" At the same time, a dialogue system using natural language processing will ask the worker, "What is the current situation?", encouraging safe behavior. In this way, the system can monitor temperature changes and human activity, and ensure worker safety by providing specific dialogue in the event of an abnormality.

[0193] Example prompt sentence:

[0194] "The temperature in the plant is reaching dangerous levels. Please issue evacuation orders and respond."

[0195] "The motion sensor has detected a worker in a dangerous area. What is the current situation?"

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] The device (a camera with a motion sensor) periodically collects data and detects the presence of people in the area. Using the video data acquired by the camera as input, it detects human movement through an image recognition algorithm and generates human presence information as output.

[0199] Step 2:

[0200] The terminal (temperature sensor) monitors temperature data at regular intervals. It receives the temperature data measured by the sensor as input, processes it to measure the current temperature, and obtains the temperature at that time as output.

[0201] Step 3:

[0202] The server collects the data obtained from both devices and monitors the presence and temperature of people. It receives the presence and temperature data as input, stores them in a database, and prepares the processing results. As output, it obtains comprehensive data showing the current situation.

[0203] Step 4:

[0204] The server evaluates the presence and temperature data and issues an audio alert if the threshold is exceeded. It uses the monitoring data as input and triggers an audio alert if there is an anomaly. As output, it generates an audio alert and a warning message.

[0205] Step 5:

[0206] The server transmits the generated alert information to an external communication device. Using the alert information as input, the server transmits this information to the external device via the Internet or wireless communication, and as output, a warning message is distributed to multiple registered terminals.

[0207] Step 6:

[0208] The server generates a prompt sentence using a generative AI model based on the alert information. The server uses the alert information as input and generates a dialogue sentence based on the AI ​​model. The dialogue sentence is obtained as output.

[0209] Step 7:

[0210] The server uses natural language processing technology to interact with the generated prompt text. It receives the prompt text and the user's response as input and processes the dialogue. As output, it obtains the dialogue results to confirm the situation and issues instructions in real time.

[0211] Step 8:

[0212] The server stores the situation information obtained from the dialogue in a database and sends appropriate instructions to the external communication device.The server uses the dialogue results as input to generate the necessary instructions, and as output, the instructions are sent to multiple registered terminals.

[0213] Step 9:

[0214] Users take safe actions according to the notifications and alerts they receive from their devices. As input, they receive messages displayed on their devices and audio alerts, and take appropriate action. As output, worker safety is ensured.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] The system of the present invention is a combination of a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine to prevent accidents involving kindergarten children and small children being left behind in vehicles. The following describes in detail the embodiments of the present invention.

[0217] Motion sensor camera operation

[0218] The terminal (camera) is equipped with a motion sensor and installed inside the vehicle. This camera periodically collects sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the motion sensor data is checked and an audio alert is issued if anyone remains inside the vehicle. The alert information is sent to a server, and notifications are sent to multiple registered terminals.

[0219] Specific examples

[0220] When the vehicle's engine is turned off and the driver removes the key, the device (camera) detects this action. If a child remains in the vehicle, the motion sensor detects their presence and issues an audio alert. The alert information is then sent to the server, and a push notification stating "There is a child remaining in the vehicle!" is sent to multiple registered smartphones.

[0221] Operation of a portable device for monitoring temperature inside a vehicle

[0222] The terminal (mobile) is a device for monitoring abnormal temperatures inside the vehicle. This mobile device periodically monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server and notified to multiple registered terminals.

[0223] Specific examples

[0224] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and issue an audio alert if it exceeds a set threshold (e.g., 35 degrees).The alert information is then sent to a server, which then sends a notification to multiple smartphones saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0225] Dialogue system and emotion engine using natural language processing technology

[0226] The server that receives the alert information activates a chatbot system using natural language processing technology. This system asks questions to the children and young children to check their situation. This invention also incorporates an emotion engine that analyzes the children's emotional state and adjusts the dialogue accordingly.

[0227] Specific examples

[0228] When an alert is issued, ChatGPT speaks to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" When the child responds, the emotion engine analyzes the response, and if the child appears anxious, it sends a reassuring message such as "It's okay, we're calling for help." If the child appears excited, it gives calming instructions such as "Try taking a slow, deep breath."

[0229] The chatbot asks additional questions based on the child's responses and emotional state, gradually understanding the situation. For example, it asks questions such as "Where are you now?" and "Are you locked in the car?" to confirm the situation. It gives instructions on how to escape or call for help, and gives specific instructions such as "Try to find the door lock button." The content of this conversation and emotional data are sent to a server, and are then sent to multiple registered smartphones as additional information.

[0230] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine.

[0231] The processing flow will be explained below.

[0232] Processing for cameras with motion sensors

[0233] Step 1:

[0234] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0235] Step 2:

[0236] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0237] Step 3:

[0238] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[0239] Step 4:

[0240] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[0241] Step 5:

[0242] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[0243] Processing a portable device that monitors the temperature inside a vehicle

[0244] Step 6:

[0245] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[0246] Step 7:

[0247] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[0248] Step 8:

[0249] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[0250] Step 9:

[0251] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[0252] Step 10:

[0253] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[0254] Dialogue processing using natural language processing technology and emotion engines

[0255] Step 11:

[0256] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[0257] Step 12:

[0258] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[0259] Step 13:

[0260] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[0261] Step 14:

[0262] The chatbot uses the child's responses and analysis data from the emotion engine to determine the child's emotional state. If the child is anxious, it sends a reassuring message, such as "It's okay, we're calling for help."

[0263] Step 15:

[0264] If a child becomes agitated, the chatbot will give them calming instructions such as, "Take slow, deep breaths and stay calm."

[0265] Step 16:

[0266] The chatbot gathers information about the situation and gives instructions on how to escape or call for help, such as sending specific instructions to the child, such as "Try to find the door lock button."

[0267] Step 17:

[0268] The chatbot transmits the content of the conversation and emotional data to the server.

[0269] Step 18:

[0270] The server then sends additional push notifications to the registered smartphone ID based on the collected situational information and emotion data, such as a notification that "A child is locked in a car. Please check immediately."

[0271] Through the above steps, the system of the present invention can ensure the safety of kindergarteners and young children through the collaboration of a camera with a motion sensor, a portable temperature monitoring device, natural language processing technology, and an emotion engine.

[0272] Example 2

[0273] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] The problem to be solved by this invention is to prevent accidents involving kindergarten children and small children being left behind in vehicles and ensure their safety. In particular, the object is to provide a system that quickly and accurately detects abnormalities and provides necessary notifications and dialogue when the vehicle engine is stopped and the key is removed, or when the temperature inside the vehicle reaches a dangerous level.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0276] In this invention, the server includes: means for receiving data from a motion sensor camera to detect the presence of a person inside the vehicle; means for issuing an audio alert if a person remains inside the vehicle when the vehicle engine is stopped and the key is removed; means for transmitting the alert information to a remote computer; means for transmitting notifications to multiple communication devices based on the transmitted alert information; means for receiving temperature data from a portable device that monitors the temperature inside the vehicle and detecting an abnormal temperature; means for issuing an audio alert when an abnormal temperature is detected; means for transmitting the alert information to the remote computer; means for conducting a dialogue using natural language processing technology; means for gathering information about the situation through the dialogue and providing instructions on how to call for help; and means for transmitting the information about the situation to the remote computer. This allows the server to quickly detect the presence of a child or toddler inside the vehicle and an abnormal temperature, and to respond immediately by transmitting a remote notification along with an audio alert. Furthermore, the server can use natural language processing technology to dialogue with the child to accurately understand the situation and reassure them.

[0277] A "camera with a human sensor" is a photographic device installed inside a vehicle and equipped with a sensor to detect human movement and presence.

[0278] The "means for receiving data" refers to a device or system for acquiring information transmitted from a camera with a motion sensor or a mobile device.

[0279] "Means for issuing an audio alert" refers to a function or device for issuing an audio warning when an abnormality is detected inside the vehicle.

[0280] The "means for transmitting alert information to an external communication device" refers to a function or system for transferring detected abnormality information to an external server or communication device using the Internet or other communication means.

[0281] The "means for sending notifications" is a system for sending warnings and notifications to multiple registered terminals based on alert information sent to an external communication device.

[0282] A "portable temperature monitoring device" is a portable device used to measure and monitor the temperature inside a vehicle.

[0283] "Means for detecting abnormal temperatures" refers to functions or systems that analyze data obtained from temperature monitoring equipment and detect abnormal temperatures that exceed a set threshold.

[0284] "Means for conducting dialogue using natural language processing technology" refers to a system that uses artificial intelligence technology to understand human language and conduct dialogue.

[0285] "Means for gathering information about the situation and providing instructions on how to seek help" is a system that understands the user's situation through dialogue and provides instructions on appropriate measures and actions.

[0286] The "means for transmitting status information to an external communication device" refers to a function or system for transferring information collected through dialogue to an external communication device or server.

[0287] As an embodiment of the present invention, a system is provided that combines various sensors and communication means to prevent accidents involving the abandonment of kindergarten children or small children in vehicles. The following describes in detail each component of this system and its operation.

[0288] Motion sensor camera operation

[0289] The device (camera with motion sensor) is installed inside the vehicle and is equipped with a sensor that detects the presence of people. Specifically, a Raspberry Pi is used, to which a motion sensor is attached. The camera periodically collects sensor data and detects the movement of people inside the vehicle. When the vehicle engine is turned off and the key is removed, the device checks the motion sensor data and issues an audio alert if anyone remains inside the vehicle. The alert information is sent to a remote computer such as AWS Lambda, which then uses Firebase Cloud Messaging to send notifications to multiple registered communication devices.

[0290] Specific examples

[0291] For example, if the vehicle engine is turned off and the driver removes the key, the device (camera) will detect this action. If a child remains in the vehicle, the human sensor will detect their presence and issue a voice alert saying, "Caution! There is someone left in the vehicle!" The alert information is then sent to the server, and a push notification saying, "Emergency! There is a child left in the vehicle!" is sent to multiple registered smartphones.

[0292] Operation of a portable device for monitoring temperature inside a vehicle

[0293] The terminal (portable device) is used to monitor the temperature inside the vehicle. Specifically, it uses an Arduino with a temperature sensor attached. The portable device periodically measures the temperature inside the vehicle and issues an audio alert if it exceeds a threshold preset by the user. The alert information is sent to a server, which then sends notifications to multiple communication devices.

[0294] Specific examples

[0295] For example, if the temperature inside a vehicle reaches 40 degrees on a hot summer day, the terminal (mobile device) detects this and issues a voice alert saying, "The temperature inside the vehicle is very high!" The alert information is then sent to the server, which then sends a notification to multiple registered smartphones saying, "Temperature warning! The temperature inside the vehicle is dangerous, please check immediately!"

[0296] Dialogue system and emotion engine operation using natural language processing technology

[0297] The server that receives the alert information uses ChatGPT to launch a dialogue system using natural language processing technology. This system asks questions to the children and toddlers to confirm their situation. This invention also incorporates an emotion engine such as Affectiva to analyze the children's emotional state and adjust the dialogue accordingly.

[0298] Specific examples

[0299] For example, when an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. Can you hear me, XX-chan?" If the child responds with "Help me!", the emotion engine will analyze the response and determine that the child is in an anxious state. The dialogue system will then send a message saying, "It's okay, I'm calling for help. Can you tell me where you are?" This information is stored on the server and sent to the parent's smartphone as additional information.

[0300] Prompt Sentence Examples

[0301] "If a child remains in the car, how do you speak to them, check in on them, and provide them with the support they need?"

[0302] In this way, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine. Each component is realized using specific hardware and software, and the results are notified to parents and guardians in real time, allowing for quick and appropriate response.

[0303] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0304] Processing for cameras with motion sensors

[0305] Step 1:

[0306] The device (camera) periodically collects data from the human presence sensor. In this case, the input is data on the presence and movement of people detected by the sensor. Specifically, a Raspberry Pi is used to acquire data from the sensor at regular intervals. The output is information on whether a person has been detected.

[0307] Step 2:

[0308] The terminal (camera) detects when the vehicle engine is stopped and the key is removed. The input is engine status information obtained from the vehicle's internal system. The output is a signal indicating that the engine has been stopped and the key has been removed. For example, the terminal can monitor the engine status in cooperation with the vehicle's system and recognize that the key has been removed.

[0309] Step 3:

[0310] The terminal (camera) checks the motion sensor data when the engine is stopped and determines whether or not there is anyone left in the vehicle. The input is the motion sensor data acquired in step 1 and the engine stop information in step 2. The output is a determination result of whether or not there is anyone left in the vehicle. Specifically, the sensor data is analyzed, and if there is anyone left, the system proceeds to the next operation.

[0311] Step 4:

[0312] If the device (camera) determines that a person remains in the vehicle, it issues a voice alert through the speaker. The input is the judgment result obtained in step 3. The output is an audio alert. For example, it may issue a message saying, "Caution! There is a person remaining in the vehicle!"

[0313] Step 5:

[0314] The device (camera) issues an audio alert and simultaneously sends the alert information to the server. The input is the audio alert information from step 4. The output is the alert information that is sent to a remote computer. Specifically, the alert information is sent to the cloud using AWS Lambda.

[0315] Step 6:

[0316] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message saying "Emergency! A child is left in the car!" is sent to the smartphone via Firebase Cloud Messaging.

[0317] Processing a portable device that monitors the temperature inside a vehicle

[0318] Step 1:

[0319] The terminal (mobile device) periodically measures the temperature inside the car. The input is data from the temperature sensor. Specifically, the temperature is periodically acquired using Arduino. The output is the current temperature information inside the car.

[0320] Step 2:

[0321] The terminal (mobile device) checks the temperature threshold set by the user in advance. The input is the threshold information set in the application. The output is the set threshold itself. Specifically, the threshold is set to 35 degrees in the application.

[0322] Step 3:

[0323] The terminal (portable device) compares the temperature data with a threshold value and detects abnormal temperatures. The inputs are the temperature information from step 1 and the threshold value information from step 2. The output is a determination result as to whether the temperature has exceeded the threshold value. Specifically, if the temperature inside the vehicle reaches 40 degrees, it is determined to be abnormal.

[0324] Step 4:

[0325] If the terminal (mobile device) detects an abnormal temperature, it issues a voice alert. The input is the judgment result of step 3. The output is an audio alert. For example, it may issue a warning saying, "The temperature inside the car is very high!"

[0326] Step 5:

[0327] The terminal (mobile device) issues a voice alert and simultaneously transmits the alert information to the server. The input is the alert information from step 4. The output is the alert information to be sent to a remote computer. Specifically, the terminal transmits information about abnormal temperature detection to the server.

[0328] Step 6:

[0329] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message is sent to the smartphone saying, "Temperature warning! The temperature inside the car is dangerous, please check immediately!"

[0330] Dialogue systems and emotion engines using natural language processing technology

[0331] Step 1:

[0332] The server launches a dialogue system using natural language processing technology based on the alert information received from the device. The input is the alert information. The output is that the dialogue system is ready to launch. Specifically, it launches a chatbot using ChatGPT.

[0333] Step 2:

[0334] The dialogue system launched by the server speaks to the kindergartener or toddler. The input is the initial question to the kindergartener. The output is a message to the kindergartener saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" Specifically, the system generates a message to gently inquire about the situation for the child.

[0335] Step 3:

[0336] The server receives the child's response and analyzes their emotional state using an emotion engine. The input is the child's response. The output is the analysis result of the child's emotional state. Specifically, the response "Help me!" is received and analyzed as an anxious state.

[0337] Step 4:

[0338] The server adjusts the dialogue content according to the emotional state. The input is the analysis result of the emotion engine. The output is an appropriately adjusted dialogue message. Specifically, it sends the message, "I'm OK, I'm calling for help. Can you tell me where you are?"

[0339] Step 5:

[0340] The server then provides more appropriate instructions based on the situational information collected through the dialogue. The input is the child's location and situation. The output is additional, more specific instructions. For example, the server might give instructions such as, "Try to find the door lock button."

[0341] Step 6:

[0342] The server notifies the registered communication devices of the content of the conversation and emotional data. The input is the content of the conversation and the emotional data. The output is a push notification with additional information. Specifically, a message is sent to the parent's smartphone saying, "Your child is asking for help. Please tell them to find the door lock button."

[0343] Through the above specific processing steps, the system of the present invention realizes a quick and accurate response to ensure the safety of kindergarten children and young children.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] In recent years, as self-driving vehicles have become more common, accidents involving kindergarten children and young children being left behind inside the vehicle have become a problem. Such accidents are particularly likely to occur when the driver leaves the vehicle, posing a serious risk to the children's lives. Furthermore, the temperature inside the vehicle may rise, putting children at risk of heatstroke. Furthermore, if a child panics inside the vehicle, it becomes difficult to respond appropriately. For these reasons, a comprehensive system to ensure the safety of kindergarten children and young children is needed in self-driving vehicles.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving data obtained from a motion sensor camera and detecting the presence of a person in the vehicle, means for issuing an audio alert if a person remains in the vehicle when the vehicle key is removed, means for transmitting the alert information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted alert information, means for conducting a dialogue using natural language processing technology, means for collecting information about the situation through the dialogue, means for analyzing responses using an emotion analysis engine and generating an appropriate message based on the responses, and means for instructing how to call for help based on the dialogue and analyzed information. This makes it possible to quickly and accurately grasp the situation of a kindergartener or young child left behind in a vehicle and safely deal with the situation.

[0348] A "camera with a human sensor" is a camera that detects the presence of people inside a vehicle and is a device that has a built-in human sensor.

[0349] A "vehicle key" is a physical or electronic key used to start or stop a vehicle engine.

[0350] "Voice alert" is a function that issues a voice warning if an abnormality is detected inside the vehicle.

[0351] The "external communication device" is a communication device installed outside the vehicle, which transmits and receives information via the Internet.

[0352] "Multiple terminals" refers to terminal devices connected to a communication network, such as smartphones and tablets owned by registered users.

[0353] "Natural language processing technology" is a technology for understanding human language and engaging in dialogue, and is a technology for analyzing and generating text.

[0354] An "emotion analysis engine" is a program that analyzes a user's emotional state based on data such as their statements and facial expressions.

[0355] "Dialogue means" refers to a means for communicating with a user, and is an interactive interface that uses voice or text.

[0356] "Instructions on how to call for help" refers to a function that provides specific instructions that children and young children need to call for help while in a vehicle.

[0357] MODE FOR CARRYING OUT THE INVENTION

[0358] The system of the present invention is designed to prevent accidents involving the abandonment of kindergarten children and young children in self-driving vehicles. The system consists of the following main components: a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, an emotion analysis engine, and a server-based system.

[0359] System Configuration

[0360] 1. Motion Sensor Camera:

[0361] The terminal, a motion sensor camera, is installed inside the vehicle and periodically collects sensor data. This camera has the function of detecting whether there is a person inside the vehicle and confirming the presence of a person when the car key is removed. If there is a person inside, it issues an audio alert and sends the corresponding alert information to the server.

[0362] 2. Portable temperature monitoring device:

[0363] The terminal, a portable temperature monitoring device, monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. This information is also sent to a server, which then sends notifications to multiple terminals.

[0364] 3. Dialogue systems using natural language processing technology:

[0365] The server activates a chatbot using natural language processing technology to converse with the children in the car. The system asks questions to check the children's status and analyzes their responses using an emotion analysis engine.

[0366] 4. Sentiment Analysis Engine:

[0367] The emotion analysis engine analyzes the emotional state of the children during the conversation and guides them to an appropriate response accordingly, a process that is important for reducing the anxiety and panic that the children may be feeling.

[0368] 5. Server and Notification System:

[0369] The server aggregates alert information and the content of conversations between the children and the nursery school, and sends notifications to multiple devices via an external communication device, allowing relevant parties to quickly grasp the situation and take appropriate action.

[0370] The specific hardware and software used

[0371] Camera and motion sensor: Using development kits such as Arduino, a dedicated camera is installed inside the vehicle.

[0372] Temperature sensor: Connect a temperature sensor to the Raspberry Pi to obtain data in real time.

[0373] Server: Uses cloud services such as AWS and Google Cloud Platform.

[0374] Natural language processing technology: A dialogue system using OpenAI's (registered trademark) GPT-3 (registered trademark).5.

[0375] Emotion analysis engine: Uses Microsoft (registered trademark) Azure (registered trademark) emotion recognition API, etc.

[0376] Specific examples

[0377] For example, suppose a vehicle's engine shuts off on a hot summer day, and the driver leaves the vehicle and removes the key. If a child is left inside the vehicle, a motion sensor camera will detect the child's presence and issue an audio alert. At the same time, a portable temperature monitoring device will detect that the temperature inside the vehicle has exceeded 35 degrees and also issue an audio alert. This alert information is sent to a server, and push notifications are sent to multiple registered smartphones, stating, "There is a child left inside the vehicle!" and "The temperature inside the vehicle is dangerous, please check immediately!"

[0378] The server then launches a dialogue system using natural language processing technology and begins asking questions to the child, such as "Hello, Mr. / Ms. X, can you hear me?" If the child responds, the emotion analysis engine analyzes the response and, if the child seems anxious, offers reassuring words like "It's okay, I'm calling for help."

[0379] Prompt Sentence Examples

[0380] Child: "Yes, I can hear you."

[0381] Dialogue system: "Hello, Mr. / Ms. X. We've detected an in-car alert. Where are you now?"

[0382] Kindergartener: "I'm near the door."

[0383] Dialogue system: "Don't worry. Someone will come to help you shortly. Try finding the door lock button."

[0384] In this way, the system of the present invention can take comprehensive measures to ensure child safety.

[0385] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0386] Step 1:

[0387] The terminal collects data inside the vehicle using a camera with a motion sensor and a portable temperature monitoring device. The motion sensor detects whether a person is inside the vehicle, and the portable temperature monitoring device monitors the temperature inside the vehicle. The data obtained from these sensors (input) is sent to the terminal as human presence and temperature information (output).

[0388] Step 2:

[0389] The terminal processes the collected data and makes a decision based on the decision criteria. Specifically, if the motion sensor detects a person inside the vehicle, or if the temperature sensor exceeds a set threshold (e.g., 35 degrees), an audio alert is issued. The terminal then issues an audio alert and sends the information to the server (input data: presence of a person, temperature information; output data: alert information).

[0390] Step 3:

[0391] The server receives alert information sent from the device. The received alert information is about the presence of people and abnormal temperatures (input). Based on this information, the server sends push notifications to multiple registered devices (output). At this stage, the alert information is delivered to the user's smartphone or tablet.

[0392] Step 4:

[0393] Upon receiving the alert information, the server activates a dialogue system using natural language processing technology. This dialogue system uses a generative AI model to generate appropriate dialogue prompts for kindergarteners and young children (input data: alert information; output data: dialogue prompts).

[0394] Step 5:

[0395] The user (a child or toddler) responds to prompts from the dialogue system. The user's response (input) is received in real time by the dialogue system. The server analyzes this response using a generative AI model and generates an appropriate reply (output).

[0396] Step 6:

[0397] The server uses an emotion analysis engine to analyze the user's emotional state from their response. Based on the analysis results, the server adjusts the content of the dialogue (input data: user's response; output data: emotion analysis results). For example, if the user shows signs of anxiety, it generates a reassuring message.

[0398] Step 7:

[0399] The server takes into account the dialogue and the results of sentiment analysis to generate specific instructions for asking for help. It generates prompts for appropriate actions along with the dialogue content and provides them to the user (input data: dialogue history, sentiment analysis results; output data: specific instructions). For example, it may give specific instructions such as "Try to find the door lock button."

[0400] Step 8:

[0401] The server sends dialogue and emotion analysis data to an external communication device and notifies multiple registered devices of the situation (input data: dialogue history, emotion analysis results; output data: situation notification). This allows the relevant parties to grasp the situation in real time.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0405] [Second embodiment]

[0406] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0407] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0413] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0417] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0418] The system of the present invention provides a means for preventing accidents in which kindergarten children or small children are left behind in vehicles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[0419] Motion sensor camera operation

[0420] The terminal (camera) is installed inside a vehicle equipped with a motion sensor. This camera periodically collects motion sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the camera checks the latest sensor data at that time and issues an audio alert if someone remains inside. In addition, by sending the alert information to a server, a notification is sent to an external communication device.

[0421] Specific examples

[0422] When the vehicle's engine is shut off and the driver removes the key, the device (camera) detects this action. At the same time, if there are any children remaining in the vehicle, the motion sensor detects their presence and issues an audio alert. In addition, the alert information is sent to the server, and a push notification saying "There are children remaining in the vehicle!" is sent to multiple registered smartphones.

[0423] Operation of a portable device for monitoring temperature inside a vehicle

[0424] The terminal (mobile phone) is used to monitor abnormal temperatures inside the vehicle. This device monitors the temperature inside the vehicle at regular intervals and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server, which then sends a notification to an external communication device.

[0425] Specific examples

[0426] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and determine that the temperature is abnormal if it exceeds 35 degrees. An audio alert will be issued and the alert information will be sent to the server. Based on this information, the server will send a push notification to the registered smartphone saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0427] Dialogue system using natural language processing technology

[0428] The server that receives the alert information activates a dialogue system (such as an AI chatbot) that uses natural language processing technology. This system asks questions to the kindergarten or toddler to confirm the situation. Based on the information obtained through this dialogue, it gives instructions on how to call for help or escape appropriately.

[0429] Specific examples

[0430] When an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," ChatGPT will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," ChatGPT will give instructions such as, "Try to find the door lock button," helping the child escape on their own. The content of this conversation is sent to the server and notified to the registered smartphone as additional information.

[0431] As a result, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and an interactive system using natural language processing technology.

[0432] The processing flow will be explained below.

[0433] Processing for cameras with motion sensors

[0434] Step 1:

[0435] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0436] Step 2:

[0437] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0438] Step 3:

[0439] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[0440] Step 4:

[0441] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[0442] Step 5:

[0443] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[0444] Processing a portable device that monitors the temperature inside a vehicle

[0445] Step 6:

[0446] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[0447] Step 7:

[0448] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[0449] Step 8:

[0450] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[0451] Step 9:

[0452] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[0453] Step 10:

[0454] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[0455] Dialogue processing using natural language processing technology

[0456] Step 11:

[0457] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[0458] Step 12:

[0459] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[0460] Step 13:

[0461] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[0462] Step 14:

[0463] Based on the child's responses, the chatbot will give them instructions on how to escape or call for help, such as "Try to find the door lock button."

[0464] Step 15:

[0465] The chatbot transmits the situation information obtained through the dialogue to the server.

[0466] Step 16:

[0467] The server then sends additional push notifications to the registered smartphone ID based on the collected situation information, such as a notification that "A child is locked in a car. Please check the situation."

[0468] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and a dialogue system using natural language processing technology.

[0469] Example 1

[0470] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0471] Accidents involving kindergarten children and young children being left behind in vehicles are a serious risk, especially in high-temperature environments. Conventional warning systems only detect the presence of people in the vehicle or abnormally high temperatures, but do not provide specific countermeasures, making immediate rescue difficult in many cases. Furthermore, warnings cannot be properly communicated to nearby adults, resulting in insufficient prevention of accidents. Therefore, there is a need for a system that can protect kindergarten children and young children by providing specific instructions and dialogue tailored to the situation.

[0472] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0473] In this invention, the server includes means for receiving data obtained from an image capture device with a motion sensor and detecting the presence of a person in the vehicle, means for issuing an audio warning if a person remains in the vehicle when the vehicle engine is stopped and the key is removed, means for transmitting the warning information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted warning information, and means for using a generative AI model to inquire about the situation and instruct appropriate responses. This makes it possible to provide not only warnings but also specific escape instructions and calls for help, enabling quick responses to prevent accidents and ensure the safety of kindergarteners and young children.

[0474] An "image capture device with a human presence sensor" is a device that is installed inside a vehicle and combines a sensor and a camera to detect the presence of people.

[0475] "Means for receiving data and detecting the presence of a person inside the vehicle" refers to a device or system that has the function of analyzing the information transmitted from the image capture device and determining whether or not a person is inside the vehicle.

[0476] The "audio warning means" refers to a system that issues an audio alert to people left in the vehicle based on the detection results.

[0477] The "means for transmitting to an external communication device" is a system for transmitting alert information to a mobile phone or an internet-connected device outside the vehicle.

[0478] The "means for sending notifications to multiple registered devices" refers to a device or system that sends alert information in the form of a push notification to specific smartphones or mobile information devices that have been registered in advance.

[0479] "Means of using generative AI models to question the situation and instruct appropriate responses" refers to an AI system that uses natural language processing technology to ask questions of people in the vehicle and instruct them on what to do.

[0480] A "portable information terminal for monitoring the temperature inside a vehicle" is a portable device equipped with a temperature sensor for periodically measuring the temperature inside the vehicle and detecting any abnormalities.

[0481] The "means for detecting abnormal temperatures" is a system that analyzes temperature data and determines whether an abnormality has occurred if the temperature exceeds a set safe range.

[0482] "Means for conducting dialogue using natural language processing technology" refers to a system that uses AI technology to simulate natural conversations with people and grasp the situation.

[0483] "Means for instructing appropriate rescue methods" is a system that provides users with appropriate actions and measures based on collected information.

[0484] The "means for transmitting warning information to an external communication device" is a system for transmitting detected abnormalities or dangerous situations to an external device via a communication network.

[0485] The "means for transmitting situation information to an external communication device" is a system that transfers situation information collected through dialogue to an external device.

[0486] The system of the present invention combines a plurality of devices and technologies to operate systematically in order to prevent accidents in which kindergarten children or small children are left behind in vehicles. Embodiments of the present invention will be described in detail below.

[0487] Motion sensor camera operation

[0488] In this system, a terminal (an image capture device with a human sensor) is installed inside the vehicle. This device is equipped with a sensor and a camera to detect the presence of people inside the vehicle, and periodically collects sensor data. For example, the camera sensor detects whether there is a person inside the vehicle every 10 seconds and stores the detection results in internal memory.

[0489] By monitoring the engine's on / off status, the device detects when the vehicle's engine is turned off and the key is removed. At this time, it checks the latest sensor data and issues an audio warning if anyone remains in the vehicle.

[0490] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends push notifications to multiple pre-registered smartphones and mobile information devices. For example, a warning message such as "There is someone remaining in the vehicle!" is sent.

[0491] Operation of a portable device for monitoring temperature inside a vehicle

[0492] The terminal (personal digital assistant) is used to monitor the temperature inside the vehicle. This terminal is equipped with a temperature sensor and measures the temperature inside the vehicle every minute. If the measured temperature exceeds a set safe range (e.g., 35 degrees), the terminal detects the abnormal temperature and issues an audio warning.

[0493] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends a push notification to multiple pre-registered smartphones and mobile information devices. For example, a message such as "The temperature inside the vehicle is dangerous. Please check immediately!" may be sent.

[0494] Dialogue system using natural language processing technology

[0495] When the server receives the alert information sent from the device, it launches a generative AI model (e.g., ChatGPT) and activates a dialogue system, which asks questions to the kindergartener or toddler to confirm the situation.

[0496] Specifically, the AI ​​chatbot will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," the AI ​​will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," the AI ​​will give instructions such as, "Try to find the door lock button," supporting the child to escape on their own.

[0497] The contents of these conversations are recorded on a server and transferred in real time to an external communication device (smartphone), allowing parents and other relevant parties to immediately understand the situation and take appropriate action.

[0498] Prompt Sentence Examples

[0499] When an alert is triggered, the following prompt is entered into the generative AI model:

[0500] "We have detected that there are children remaining in the vehicle. Please begin a dialogue with the children to confirm their situation."

[0501] This enables the AI ​​model to generate appropriate dialogue depending on the situation, supporting young children and kindergarteners.

[0502] As described above, the system of the present invention can ensure the safety of kindergarten children and young children inside vehicles by combining a camera with a human sensor, a temperature monitoring mobile terminal, and a dialogue system using natural language processing technology.

[0503] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] The terminal (an image capture device with a motion sensor) collects data from the motion sensor every 10 seconds to detect the presence of people inside the vehicle.

[0506] Specific operation: When the sensor detects the presence of a person, the data (input) is stored in the camera's internal memory. The stored data is output.

[0507] Step 2:

[0508] The terminal monitors the vehicle's engine status and detects when the engine is turned off and the key is removed.

[0509] Specific operation: The system monitors the engine on / off status and key removal timing (input), and makes an alert decision based on that information. The timing of key removal and engine status are output.

[0510] Step 3:

[0511] When the key is removed, the terminal checks the latest motion sensor data to determine whether anyone remains in the vehicle.

[0512] Specific operation: Analyzes the latest stored sensor data (input), and if a person is detected, generates and issues an audio alert. The result of whether or not an alert was issued is output.

[0513] Step 4:

[0514] The terminal transmits the issued alert information to the server.

[0515] Specific operation: Alert information (input) is processed to be sent to an external communication device, and then sent to a server via a network. The server outputs a message indicating that it has received the alert information.

[0516] Step 5:

[0517] The server sends a notification to the registered terminals based on the received alert information.

[0518] Specific operation: The server analyzes the received alert data (input) and sends a push notification to the registered smartphone or information terminal. The result of the notification being sent is output.

[0519] Step 6:

[0520] The terminal (personal digital assistant) periodically monitors the temperature inside the vehicle.

[0521] Specific operation: The temperature sensor measures the temperature inside the vehicle every minute, stores the data (input) in the internal memory, and outputs the measurement results.

[0522] Step 7:

[0523] The terminal analyzes the measurement data and detects an abnormal temperature if the temperature exceeds a set temperature threshold.

[0524] Specific operation: Analyzes the measured in-car temperature data (input) and determines that an abnormality has occurred if the temperature exceeds a preset temperature threshold (e.g., 35°C). The abnormal temperature detection result is output.

[0525] Step 8:

[0526] The device will issue an audio warning if it detects an abnormal temperature.

[0527] Specific operation: Based on the abnormal temperature detection result (input), an audio alert is generated and output.

[0528] Step 9:

[0529] The terminal transmits the warning information to the server.

[0530] Specific operation: The abnormal temperature warning information (input) is processed and sent to the server via the network. The server outputs the information that it has received the warning information.

[0531] Step 10:

[0532] The server sends a notification to multiple registered terminals based on the received abnormal temperature warning information.

[0533] Specific operation: The server analyzes the abnormal temperature data (input) and sends a push notification to registered devices such as smartphones. The notification result is output.

[0534] Step 11:

[0535] The server launches a generative AI model (e.g., ChatGPT) based on the received alert information.

[0536] Specific operation: A prompt sentence (input) saying "There are children remaining in the vehicle. Please check with the children" is sent to the AI ​​model, and a dialogue scenario is generated. The generated dialogue scenario is output.

[0537] Step 12:

[0538] The server then has the generative AI model begin asking questions to the children based on the dialogue scenario.

[0539] Specific operation: Based on the prompt sentence (input), the AI ​​starts a dialogue saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" The progress of the dialogue is output.

[0540] Step 13:

[0541] The server records information obtained through the interaction and sends notifications to external communication devices.

[0542] Specific operation: The situation information (input) collected through the dialogue is stored in a database and transferred to a smartphone in real time. The notification result is output.

[0543] As described above, the various terminals and servers of this system work in cooperation to provide a quick and effective response to protect kindergarten children and young children inside vehicles.

[0544] (Application example 1)

[0545] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0546] Currently, safety management in factories places a heavy burden on workers. It is particularly difficult to grasp and respond to situations when working in high temperatures or approaching dangerous areas, making it necessary to ensure safety quickly and accurately. Furthermore, existing systems do not adequately check the detailed situation when an alert is issued, potentially threatening worker safety. Therefore, a comprehensive system is needed that can efficiently monitor temperatures in factories, detect movement using human sensors, and check the situation and issue instructions in the event of an abnormality.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0548] In this invention, the server includes: means for receiving data obtained from a motion sensor camera and detecting the presence of people within the area; means for issuing an audio alert if certain conditions are met and a person remains; means for transmitting the alert information to an external communication device; means for sending notifications to multiple registered terminals based on the transmitted alert information; means for conducting a dialogue based on the alert information using natural language processing technology; and means for gathering information on the situation through the dialogue and issuing appropriate instructions. This enables prompt and accurate safety management within the factory. Temperature changes and human activity can be monitored, and specific dialogue can be provided to ensure worker safety in the event of an abnormality.

[0549] A "camera with a human sensor" is a device that detects human movement and collects that information through the camera.

[0550] "Means for detecting the presence of people within an area" refers to technology that uses sensors or cameras to confirm that people are present within a specific area.

[0551] "A means for issuing an audio alert" refers to a device or system that issues an audio warning when an abnormality or emergency is detected.

[0552] "Means for transmitting to an external communication device" refers to technology for transmitting collected data and alert information to other external devices via the Internet or wireless communication.

[0553] The "means for sending notifications to multiple registered terminals" is a system for simultaneously sending notifications to multiple devices, such as smartphones and tablets, that have been registered in advance.

[0554] "Means of conducting dialogue using natural language processing technology" refers to technology that uses AI and machine learning to understand human language and conduct dialogue.

[0555] "Means of gathering information about the situation and issuing appropriate instructions" refers to technology that uses dialogue and sensor data to grasp the situation on-site and issue necessary instructions to people.

[0556] The "means for receiving temperature data and detecting abnormal temperatures" is a technology for receiving data from a temperature sensor and detecting that the data is outside a safe range.

[0557] "Means for generating prompt sentences using a generative AI model" refers to a technology that uses an AI learning model to automatically generate appropriate dialogue sentences or instructions based on specified content.

[0558] System configuration

[0559] This invention is a system that includes a motion sensor camera, a portable temperature monitoring device, a dialogue system using natural language processing technology, and a generative AI model. This system can perform fast and accurate monitoring to ensure the safety of workers in factories.

[0560] Hardware and Software

[0561] Hardware:

[0562] Smart glasses (e.g., Google Glass): worn by workers to display alert notifications and interactions.

[0563] Server (e.g. AWS EC2 instance): Collects data, processes it, and sends notifications.

[0564] Temperature sensor (e.g. DHT22): Monitors the temperature inside the factory.

[0565] Camera with motion sensor (e.g. Raspberry Pi Camera Module): Monitors activity within the factory.

[0566] software:

[0567] Natural language processing technology (e.g. ChatGPT): Used to communicate with workers and check the status.

[0568] Generative AI model: Used to generate prompts.

[0569] Data processing and calculation

[0570] The server receives data from cameras equipped with motion sensors and detects the presence of people within the area. Based on this information, if a person meets certain conditions (for example, if they approach a dangerous area), it issues an audio alert. It also continuously receives data from temperature sensors and detects abnormal temperatures when they exceed a set threshold. It then transmits this alert information to an external communications device, which then sends notifications to multiple registered devices.

[0571] Furthermore, a generative AI model is used to generate prompts based on the alert information, and natural language processing technology is used to conduct dialogue, confirming the worker's status in real time and issuing necessary instructions. For example, when a voice alert occurs, the server can use ChatGPT to generate a prompt such as "Please tell us your current situation" and begin dialogue.

[0572] Specific examples

[0573] If a worker working in a factory machine room experiences a sudden temperature rise and reaches 40 degrees, the smart glasses will issue a warning and send a push notification saying, "The temperature in the factory is at a dangerous level. Please evacuate!" At the same time, a dialogue system using natural language processing will ask the worker, "What is the current situation?", encouraging safe behavior. In this way, the system can monitor temperature changes and human activity, and ensure worker safety by providing specific dialogue in the event of an abnormality.

[0574] Example prompt sentence:

[0575] "The temperature in the plant is reaching dangerous levels. Please issue evacuation orders and respond."

[0576] "The motion sensor has detected a worker in a dangerous area. What is the current situation?"

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1:

[0579] The device (a camera with a motion sensor) periodically collects data and detects the presence of people in the area. Using the video data acquired by the camera as input, it detects human movement through an image recognition algorithm and generates human presence information as output.

[0580] Step 2:

[0581] The terminal (temperature sensor) monitors temperature data at regular intervals. It receives the temperature data measured by the sensor as input, processes it to measure the current temperature, and obtains the temperature at that time as output.

[0582] Step 3:

[0583] The server collects the data obtained from both devices and monitors the presence and temperature of people. It receives the presence and temperature data as input, stores them in a database, and prepares the processing results. As output, it obtains comprehensive data showing the current situation.

[0584] Step 4:

[0585] The server evaluates the presence and temperature data and issues an audio alert if the threshold is exceeded. It uses the monitoring data as input and triggers an audio alert if there is an anomaly. As output, it generates an audio alert and a warning message.

[0586] Step 5:

[0587] The server transmits the generated alert information to an external communication device. Using the alert information as input, the server transmits this information to the external device via the Internet or wireless communication, and as output, a warning message is distributed to multiple registered terminals.

[0588] Step 6:

[0589] The server generates a prompt sentence using a generative AI model based on the alert information. The server uses the alert information as input and generates a dialogue sentence based on the AI ​​model. The dialogue sentence is obtained as output.

[0590] Step 7:

[0591] The server uses natural language processing technology to interact with the generated prompt text. It receives the prompt text and the user's response as input and processes the dialogue. As output, it obtains the dialogue results to confirm the situation and issues instructions in real time.

[0592] Step 8:

[0593] The server stores the situation information obtained from the dialogue in a database and sends appropriate instructions to the external communication device.The server uses the dialogue results as input to generate the necessary instructions, and as output, the instructions are sent to multiple registered terminals.

[0594] Step 9:

[0595] Users take safe actions according to the notifications and alerts they receive from their devices. As input, they receive messages displayed on their devices and audio alerts, and take appropriate action. As output, worker safety is ensured.

[0596] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0597] The system of the present invention is a combination of a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine to prevent accidents involving kindergarten children and small children being left behind in vehicles. The following describes in detail the embodiments of the present invention.

[0598] Motion sensor camera operation

[0599] The terminal (camera) is equipped with a motion sensor and installed inside the vehicle. This camera periodically collects sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the motion sensor data is checked and an audio alert is issued if anyone remains inside the vehicle. The alert information is sent to a server, and notifications are sent to multiple registered terminals.

[0600] Specific examples

[0601] When the vehicle's engine is turned off and the driver removes the key, the device (camera) detects this action. If a child remains in the vehicle, the motion sensor detects their presence and issues an audio alert. The alert information is then sent to the server, and a push notification stating "There is a child remaining in the vehicle!" is sent to multiple registered smartphones.

[0602] Operation of a portable device for monitoring temperature inside a vehicle

[0603] The terminal (mobile) is a device for monitoring abnormal temperatures inside the vehicle. This mobile device periodically monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server and notified to multiple registered terminals.

[0604] Specific examples

[0605] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and issue an audio alert if it exceeds a set threshold (e.g., 35 degrees).The alert information is then sent to a server, which then sends a notification to multiple smartphones saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0606] Dialogue system and emotion engine using natural language processing technology

[0607] The server that receives the alert information activates a chatbot system using natural language processing technology. This system asks questions to the children and young children to check their situation. This invention also incorporates an emotion engine that analyzes the children's emotional state and adjusts the dialogue accordingly.

[0608] Specific examples

[0609] When an alert is issued, ChatGPT speaks to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" When the child responds, the emotion engine analyzes the response, and if the child appears anxious, it sends a reassuring message such as "It's okay, we're calling for help." If the child appears excited, it gives calming instructions such as "Try taking a slow, deep breath."

[0610] The chatbot asks additional questions based on the child's responses and emotional state, gradually understanding the situation. For example, it asks questions such as "Where are you now?" and "Are you locked in the car?" to confirm the situation. It gives instructions on how to escape or call for help, and gives specific instructions such as "Try to find the door lock button." The content of this conversation and emotional data are sent to a server, and are then sent to multiple registered smartphones as additional information.

[0611] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine.

[0612] The processing flow will be explained below.

[0613] Processing for cameras with motion sensors

[0614] Step 1:

[0615] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0616] Step 2:

[0617] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0618] Step 3:

[0619] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[0620] Step 4:

[0621] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[0622] Step 5:

[0623] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[0624] Processing a portable device that monitors the temperature inside a vehicle

[0625] Step 6:

[0626] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[0627] Step 7:

[0628] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[0629] Step 8:

[0630] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[0631] Step 9:

[0632] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[0633] Step 10:

[0634] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[0635] Dialogue processing using natural language processing technology and emotion engines

[0636] Step 11:

[0637] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[0638] Step 12:

[0639] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[0640] Step 13:

[0641] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[0642] Step 14:

[0643] The chatbot uses the child's responses and analysis data from the emotion engine to determine the child's emotional state. If the child is anxious, it sends a reassuring message, such as "It's okay, we're calling for help."

[0644] Step 15:

[0645] If a child becomes agitated, the chatbot will give them calming instructions such as, "Take slow, deep breaths and stay calm."

[0646] Step 16:

[0647] The chatbot gathers information about the situation and gives instructions on how to escape or call for help, such as sending specific instructions to the child, such as "Try to find the door lock button."

[0648] Step 17:

[0649] The chatbot transmits the content of the conversation and emotional data to the server.

[0650] Step 18:

[0651] The server then sends additional push notifications to the registered smartphone ID based on the collected situational information and emotion data, such as a notification that "A child is locked in a car. Please check immediately."

[0652] Through the above steps, the system of the present invention can ensure the safety of kindergarteners and young children through the collaboration of a camera with a motion sensor, a portable temperature monitoring device, natural language processing technology, and an emotion engine.

[0653] Example 2

[0654] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0655] The problem to be solved by this invention is to prevent accidents involving kindergarten children and small children being left behind in vehicles and ensure their safety. In particular, the object is to provide a system that quickly and accurately detects abnormalities and provides necessary notifications and dialogue when the vehicle engine is stopped and the key is removed, or when the temperature inside the vehicle reaches a dangerous level.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0657] In this invention, the server includes: means for receiving data from a motion sensor camera to detect the presence of a person inside the vehicle; means for issuing an audio alert if a person remains inside the vehicle when the vehicle engine is stopped and the key is removed; means for transmitting the alert information to a remote computer; means for transmitting notifications to multiple communication devices based on the transmitted alert information; means for receiving temperature data from a portable device that monitors the temperature inside the vehicle and detecting an abnormal temperature; means for issuing an audio alert when an abnormal temperature is detected; means for transmitting the alert information to the remote computer; means for conducting a dialogue using natural language processing technology; means for gathering information about the situation through the dialogue and providing instructions on how to call for help; and means for transmitting the information about the situation to the remote computer. This allows the server to quickly detect the presence of a child or toddler inside the vehicle and an abnormal temperature, and to respond immediately by transmitting a remote notification along with an audio alert. Furthermore, the server can use natural language processing technology to dialogue with the child to accurately understand the situation and reassure them.

[0658] A "camera with a human sensor" is a photographic device installed inside a vehicle and equipped with a sensor to detect human movement and presence.

[0659] The "means for receiving data" refers to a device or system for acquiring information transmitted from a camera with a motion sensor or a mobile device.

[0660] "Means for issuing an audio alert" refers to a function or device for issuing an audio warning when an abnormality is detected inside the vehicle.

[0661] The "means for transmitting alert information to an external communication device" refers to a function or system for transferring detected abnormality information to an external server or communication device using the Internet or other communication means.

[0662] The "means for sending notifications" is a system for sending warnings and notifications to multiple registered terminals based on alert information sent to an external communication device.

[0663] A "portable temperature monitoring device" is a portable device used to measure and monitor the temperature inside a vehicle.

[0664] "Means for detecting abnormal temperatures" refers to functions or systems that analyze data obtained from temperature monitoring equipment and detect abnormal temperatures that exceed a set threshold.

[0665] "Means for conducting dialogue using natural language processing technology" refers to a system that uses artificial intelligence technology to understand human language and conduct dialogue.

[0666] "Means for gathering information about the situation and providing instructions on how to seek help" is a system that understands the user's situation through dialogue and provides instructions on appropriate measures and actions.

[0667] The "means for transmitting status information to an external communication device" refers to a function or system for transferring information collected through dialogue to an external communication device or server.

[0668] As an embodiment of the present invention, a system is provided that combines various sensors and communication means to prevent accidents involving the abandonment of kindergarten children or small children in vehicles. The following describes in detail each component of this system and its operation.

[0669] Motion sensor camera operation

[0670] The device (camera with motion sensor) is installed inside the vehicle and is equipped with a sensor that detects the presence of people. Specifically, a Raspberry Pi is used, to which a motion sensor is attached. The camera periodically collects sensor data and detects the movement of people inside the vehicle. When the vehicle engine is turned off and the key is removed, the device checks the motion sensor data and issues an audio alert if anyone remains inside the vehicle. The alert information is sent to a remote computer such as AWS Lambda, which then uses Firebase Cloud Messaging to send notifications to multiple registered communication devices.

[0671] Specific examples

[0672] For example, if the vehicle engine is turned off and the driver removes the key, the device (camera) will detect this action. If a child remains in the vehicle, the human sensor will detect their presence and issue a voice alert saying, "Caution! There is someone left in the vehicle!" The alert information is then sent to the server, and a push notification saying, "Emergency! There is a child left in the vehicle!" is sent to multiple registered smartphones.

[0673] Operation of a portable device for monitoring temperature inside a vehicle

[0674] The terminal (portable device) is used to monitor the temperature inside the vehicle. Specifically, it uses an Arduino with a temperature sensor attached. The portable device periodically measures the temperature inside the vehicle and issues an audio alert if it exceeds a threshold preset by the user. The alert information is sent to a server, which then sends notifications to multiple communication devices.

[0675] Specific examples

[0676] For example, if the temperature inside a vehicle reaches 40 degrees on a hot summer day, the terminal (mobile device) detects this and issues a voice alert saying, "The temperature inside the vehicle is very high!" The alert information is then sent to the server, which then sends a notification to multiple registered smartphones saying, "Temperature warning! The temperature inside the vehicle is dangerous, please check immediately!"

[0677] Dialogue system and emotion engine operation using natural language processing technology

[0678] The server that receives the alert information uses ChatGPT to launch a dialogue system using natural language processing technology. This system asks questions to the children and toddlers to confirm their situation. This invention also incorporates an emotion engine such as Affectiva to analyze the children's emotional state and adjust the dialogue accordingly.

[0679] Specific examples

[0680] For example, when an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. Can you hear me, XX-chan?" If the child responds with "Help me!", the emotion engine will analyze the response and determine that the child is in an anxious state. The dialogue system will then send a message saying, "It's okay, I'm calling for help. Can you tell me where you are?" This information is stored on the server and sent to the parent's smartphone as additional information.

[0681] Prompt Sentence Examples

[0682] "If a child remains in the car, how do you speak to them, check in on them, and provide them with the support they need?"

[0683] In this way, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine. Each component is realized using specific hardware and software, and the results are notified to parents and guardians in real time, allowing for quick and appropriate response.

[0684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0685] Processing for cameras with motion sensors

[0686] Step 1:

[0687] The device (camera) periodically collects data from the human presence sensor. In this case, the input is data on the presence and movement of people detected by the sensor. Specifically, a Raspberry Pi is used to acquire data from the sensor at regular intervals. The output is information on whether a person has been detected.

[0688] Step 2:

[0689] The terminal (camera) detects when the vehicle engine is stopped and the key is removed. The input is engine status information obtained from the vehicle's internal system. The output is a signal indicating that the engine has been stopped and the key has been removed. For example, the terminal can monitor the engine status in cooperation with the vehicle's system and recognize that the key has been removed.

[0690] Step 3:

[0691] The terminal (camera) checks the motion sensor data when the engine is stopped and determines whether or not there is anyone left in the vehicle. The input is the motion sensor data acquired in step 1 and the engine stop information in step 2. The output is a determination result of whether or not there is anyone left in the vehicle. Specifically, the sensor data is analyzed, and if there is anyone left, the system proceeds to the next operation.

[0692] Step 4:

[0693] If the device (camera) determines that a person remains in the vehicle, it issues a voice alert through the speaker. The input is the judgment result obtained in step 3. The output is an audio alert. For example, it may issue a message saying, "Caution! There is a person remaining in the vehicle!"

[0694] Step 5:

[0695] The device (camera) issues an audio alert and simultaneously sends the alert information to the server. The input is the audio alert information from step 4. The output is the alert information that is sent to a remote computer. Specifically, the alert information is sent to the cloud using AWS Lambda.

[0696] Step 6:

[0697] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message saying "Emergency! A child is left in the car!" is sent to the smartphone via Firebase Cloud Messaging.

[0698] Processing a portable device that monitors the temperature inside a vehicle

[0699] Step 1:

[0700] The terminal (mobile device) periodically measures the temperature inside the car. The input is data from the temperature sensor. Specifically, the temperature is periodically acquired using Arduino. The output is the current temperature information inside the car.

[0701] Step 2:

[0702] The terminal (mobile device) checks the temperature threshold set by the user in advance. The input is the threshold information set in the application. The output is the set threshold itself. Specifically, the threshold is set to 35 degrees in the application.

[0703] Step 3:

[0704] The terminal (portable device) compares the temperature data with a threshold value and detects abnormal temperatures. The inputs are the temperature information from step 1 and the threshold value information from step 2. The output is a determination result as to whether the temperature has exceeded the threshold value. Specifically, if the temperature inside the vehicle reaches 40 degrees, it is determined to be abnormal.

[0705] Step 4:

[0706] If the terminal (mobile device) detects an abnormal temperature, it issues a voice alert. The input is the judgment result of step 3. The output is an audio alert. For example, it may issue a warning saying, "The temperature inside the car is very high!"

[0707] Step 5:

[0708] The terminal (mobile device) issues a voice alert and simultaneously transmits the alert information to the server. The input is the alert information from step 4. The output is the alert information to be sent to a remote computer. Specifically, the terminal transmits information about abnormal temperature detection to the server.

[0709] Step 6:

[0710] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message is sent to the smartphone saying, "Temperature warning! The temperature inside the car is dangerous, please check immediately!"

[0711] Dialogue systems and emotion engines using natural language processing technology

[0712] Step 1:

[0713] The server launches a dialogue system using natural language processing technology based on the alert information received from the device. The input is the alert information. The output is that the dialogue system is ready to launch. Specifically, it launches a chatbot using ChatGPT.

[0714] Step 2:

[0715] The dialogue system launched by the server speaks to the kindergartener or toddler. The input is the initial question to the kindergartener. The output is a message to the kindergartener saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" Specifically, the system generates a message to gently inquire about the situation for the child.

[0716] Step 3:

[0717] The server receives the child's response and analyzes their emotional state using an emotion engine. The input is the child's response. The output is the analysis result of the child's emotional state. Specifically, the response "Help me!" is received and analyzed as an anxious state.

[0718] Step 4:

[0719] The server adjusts the dialogue content according to the emotional state. The input is the analysis result of the emotion engine. The output is an appropriately adjusted dialogue message. Specifically, it sends the message, "I'm OK, I'm calling for help. Can you tell me where you are?"

[0720] Step 5:

[0721] The server then provides more appropriate instructions based on the situational information collected through the dialogue. The input is the child's location and situation. The output is additional, more specific instructions. For example, the server might give instructions such as, "Try to find the door lock button."

[0722] Step 6:

[0723] The server notifies the registered communication devices of the content of the conversation and emotional data. The input is the content of the conversation and the emotional data. The output is a push notification with additional information. Specifically, a message is sent to the parent's smartphone saying, "Your child is asking for help. Please tell them to find the door lock button."

[0724] Through the above specific processing steps, the system of the present invention realizes a quick and accurate response to ensure the safety of kindergarten children and young children.

[0725] (Application example 2)

[0726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] In recent years, as self-driving vehicles have become more common, accidents involving kindergarten children and young children being left behind inside the vehicle have become a problem. Such accidents are particularly likely to occur when the driver leaves the vehicle, posing a serious risk to the children's lives. Furthermore, the temperature inside the vehicle may rise, putting children at risk of heatstroke. Furthermore, if a child panics inside the vehicle, it becomes difficult to respond appropriately. For these reasons, a comprehensive system to ensure the safety of kindergarten children and young children is needed in self-driving vehicles.

[0728] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving data obtained from a motion sensor camera and detecting the presence of a person in the vehicle, means for issuing an audio alert if a person remains in the vehicle when the vehicle key is removed, means for transmitting the alert information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted alert information, means for conducting a dialogue using natural language processing technology, means for collecting information about the situation through the dialogue, means for analyzing responses using an emotion analysis engine and generating an appropriate message based on the responses, and means for instructing how to call for help based on the dialogue and analyzed information. This makes it possible to quickly and accurately grasp the situation of a kindergartener or young child left behind in a vehicle and safely deal with the situation.

[0729] A "camera with a human sensor" is a camera that detects the presence of people inside a vehicle and is a device that has a built-in human sensor.

[0730] A "vehicle key" is a physical or electronic key used to start or stop a vehicle engine.

[0731] "Voice alert" is a function that issues a voice warning if an abnormality is detected inside the vehicle.

[0732] The "external communication device" is a communication device installed outside the vehicle, which transmits and receives information via the Internet.

[0733] "Multiple terminals" refers to terminal devices connected to a communication network, such as smartphones and tablets owned by registered users.

[0734] "Natural language processing technology" is a technology for understanding human language and engaging in dialogue, and is a technology for analyzing and generating text.

[0735] An "emotion analysis engine" is a program that analyzes a user's emotional state based on data such as their statements and facial expressions.

[0736] "Dialogue means" refers to a means for communicating with a user, and is an interactive interface that uses voice or text.

[0737] "Instructions on how to call for help" refers to a function that provides specific instructions that children and young children need to call for help while in a vehicle.

[0738] MODE FOR CARRYING OUT THE INVENTION

[0739] The system of the present invention is designed to prevent accidents involving the abandonment of kindergarten children and young children in self-driving vehicles. The system consists of the following main components: a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, an emotion analysis engine, and a server-based system.

[0740] System Configuration

[0741] 1. Motion Sensor Camera:

[0742] The terminal, a motion sensor camera, is installed inside the vehicle and periodically collects sensor data. This camera has the function of detecting whether there is a person inside the vehicle and confirming the presence of a person when the car key is removed. If there is a person inside, it issues an audio alert and sends the corresponding alert information to the server.

[0743] 2. Portable temperature monitoring device:

[0744] The terminal, a portable temperature monitoring device, monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. This information is also sent to a server, which then sends notifications to multiple terminals.

[0745] 3. Dialogue systems using natural language processing technology:

[0746] The server activates a chatbot using natural language processing technology to converse with the children in the car. The system asks questions to check the children's status and analyzes their responses using an emotion analysis engine.

[0747] 4. Sentiment Analysis Engine:

[0748] The emotion analysis engine analyzes the emotional state of the children during the conversation and guides them to an appropriate response accordingly, a process that is important for reducing the anxiety and panic that the children may be feeling.

[0749] 5. Server and Notification System:

[0750] The server aggregates alert information and the content of conversations between the children and the nursery school, and sends notifications to multiple devices via an external communication device, allowing relevant parties to quickly grasp the situation and take appropriate action.

[0751] The specific hardware and software used

[0752] Camera and motion sensor: Using development kits such as Arduino, a dedicated camera is installed inside the vehicle.

[0753] Temperature sensor: Connect a temperature sensor to the Raspberry Pi to obtain data in real time.

[0754] Server: Uses cloud services such as AWS and Google Cloud Platform.

[0755] Natural language processing technology: A dialogue system using OpenAI's GPT-3.5.

[0756] Sentiment analysis engine: Uses Microsoft Azure's emotion recognition API, etc.

[0757] Specific examples

[0758] For example, suppose a vehicle's engine shuts off on a hot summer day, and the driver leaves the vehicle and removes the key. If a child is left inside the vehicle, a motion sensor camera will detect the child's presence and issue an audio alert. At the same time, a portable temperature monitoring device will detect that the temperature inside the vehicle has exceeded 35 degrees and also issue an audio alert. This alert information is sent to a server, and push notifications are sent to multiple registered smartphones, stating, "There is a child left inside the vehicle!" and "The temperature inside the vehicle is dangerous, please check immediately!"

[0759] The server then launches a dialogue system using natural language processing technology and begins asking questions to the child, such as "Hello, Mr. / Ms. X, can you hear me?" If the child responds, the emotion analysis engine analyzes the response and, if the child seems anxious, offers reassuring words like "It's okay, I'm calling for help."

[0760] Prompt Sentence Examples

[0761] Child: "Yes, I can hear you."

[0762] Dialogue system: "Hello, Mr. / Ms. X. We've detected an in-car alert. Where are you now?"

[0763] Kindergartener: "I'm near the door."

[0764] Dialogue system: "Don't worry. Someone will come to help you shortly. Try finding the door lock button."

[0765] In this way, the system of the present invention can take comprehensive measures to ensure child safety.

[0766] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0767] Step 1:

[0768] The terminal collects data inside the vehicle using a camera with a motion sensor and a portable temperature monitoring device. The motion sensor detects whether a person is inside the vehicle, and the portable temperature monitoring device monitors the temperature inside the vehicle. The data obtained from these sensors (input) is sent to the terminal as human presence and temperature information (output).

[0769] Step 2:

[0770] The terminal processes the collected data and makes a decision based on the decision criteria. Specifically, if the motion sensor detects a person inside the vehicle, or if the temperature sensor exceeds a set threshold (e.g., 35 degrees), an audio alert is issued. The terminal then issues an audio alert and sends the information to the server (input data: presence of a person, temperature information; output data: alert information).

[0771] Step 3:

[0772] The server receives alert information sent from the device. The received alert information is about the presence of people and abnormal temperatures (input). Based on this information, the server sends push notifications to multiple registered devices (output). At this stage, the alert information is delivered to the user's smartphone or tablet.

[0773] Step 4:

[0774] Upon receiving the alert information, the server activates a dialogue system using natural language processing technology. This dialogue system uses a generative AI model to generate appropriate dialogue prompts for kindergarteners and young children (input data: alert information; output data: dialogue prompts).

[0775] Step 5:

[0776] The user (a child or toddler) responds to prompts from the dialogue system. The user's response (input) is received in real time by the dialogue system. The server analyzes this response using a generative AI model and generates an appropriate reply (output).

[0777] Step 6:

[0778] The server uses an emotion analysis engine to analyze the user's emotional state from their response. Based on the analysis results, the server adjusts the content of the dialogue (input data: user's response; output data: emotion analysis results). For example, if the user shows signs of anxiety, it generates a reassuring message.

[0779] Step 7:

[0780] The server takes into account the dialogue and the results of sentiment analysis to generate specific instructions for asking for help. It generates prompts for appropriate actions along with the dialogue content and provides them to the user (input data: dialogue history, sentiment analysis results; output data: specific instructions). For example, it may give specific instructions such as "Try to find the door lock button."

[0781] Step 8:

[0782] The server sends dialogue and emotion analysis data to an external communication device and notifies multiple registered devices of the situation (input data: dialogue history, emotion analysis results; output data: situation notification). This allows the relevant parties to grasp the situation in real time.

[0783] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0784] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0785] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0786] [Third embodiment]

[0787] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0788] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0789] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0790] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0791] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0792] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0793] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0794] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0795] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0796] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0797] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0798] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0799] The system of the present invention provides a means for preventing accidents in which kindergarten children or small children are left behind in vehicles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[0800] Motion sensor camera operation

[0801] The terminal (camera) is installed inside a vehicle equipped with a motion sensor. This camera periodically collects motion sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the camera checks the latest sensor data at that time and issues an audio alert if someone remains inside. In addition, by sending the alert information to a server, a notification is sent to an external communication device.

[0802] Specific examples

[0803] When the vehicle's engine is shut off and the driver removes the key, the device (camera) detects this action. At the same time, if there are any children remaining in the vehicle, the motion sensor detects their presence and issues an audio alert. In addition, the alert information is sent to the server, and a push notification saying "There are children remaining in the vehicle!" is sent to multiple registered smartphones.

[0804] Operation of a portable device for monitoring temperature inside a vehicle

[0805] The terminal (mobile phone) is used to monitor abnormal temperatures inside the vehicle. This device monitors the temperature inside the vehicle at regular intervals and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server, which then sends a notification to an external communication device.

[0806] Specific examples

[0807] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and determine that the temperature is abnormal if it exceeds 35 degrees. An audio alert will be issued and the alert information will be sent to the server. Based on this information, the server will send a push notification to the registered smartphone saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0808] Dialogue system using natural language processing technology

[0809] The server that receives the alert information activates a dialogue system (such as an AI chatbot) that uses natural language processing technology. This system asks questions to the kindergarten or toddler to confirm the situation. Based on the information obtained through this dialogue, it gives instructions on how to call for help or escape appropriately.

[0810] Specific examples

[0811] When an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," ChatGPT will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," ChatGPT will give instructions such as, "Try to find the door lock button," helping the child escape on their own. The content of this conversation is sent to the server and notified to the registered smartphone as additional information.

[0812] As a result, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and an interactive system using natural language processing technology.

[0813] The processing flow will be explained below.

[0814] Processing for cameras with motion sensors

[0815] Step 1:

[0816] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0817] Step 2:

[0818] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0819] Step 3:

[0820] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[0821] Step 4:

[0822] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[0823] Step 5:

[0824] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[0825] Processing a portable device that monitors the temperature inside a vehicle

[0826] Step 6:

[0827] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[0828] Step 7:

[0829] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[0830] Step 8:

[0831] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[0832] Step 9:

[0833] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[0834] Step 10:

[0835] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[0836] Dialogue processing using natural language processing technology

[0837] Step 11:

[0838] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[0839] Step 12:

[0840] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[0841] Step 13:

[0842] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[0843] Step 14:

[0844] Based on the child's responses, the chatbot will give them instructions on how to escape or call for help, such as "Try to find the door lock button."

[0845] Step 15:

[0846] The chatbot transmits the situation information obtained through the dialogue to the server.

[0847] Step 16:

[0848] The server then sends additional push notifications to the registered smartphone ID based on the collected situation information, such as a notification that "A child is locked in a car. Please check the situation."

[0849] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and a dialogue system using natural language processing technology.

[0850] Example 1

[0851] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0852] Accidents involving kindergarten children and young children being left behind in vehicles are a serious risk, especially in high-temperature environments. Conventional warning systems only detect the presence of people in the vehicle or abnormally high temperatures, but do not provide specific countermeasures, making immediate rescue difficult in many cases. Furthermore, warnings cannot be properly communicated to nearby adults, resulting in insufficient prevention of accidents. Therefore, there is a need for a system that can protect kindergarten children and young children by providing specific instructions and dialogue tailored to the situation.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0854] In this invention, the server includes means for receiving data obtained from an image capture device with a motion sensor and detecting the presence of a person in the vehicle, means for issuing an audio warning if a person remains in the vehicle when the vehicle engine is stopped and the key is removed, means for transmitting the warning information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted warning information, and means for using a generative AI model to inquire about the situation and instruct appropriate responses. This makes it possible to provide not only warnings but also specific escape instructions and calls for help, enabling quick responses to prevent accidents and ensure the safety of kindergarteners and young children.

[0855] An "image capture device with a human presence sensor" is a device that is installed inside a vehicle and combines a sensor and a camera to detect the presence of people.

[0856] "Means for receiving data and detecting the presence of a person inside the vehicle" refers to a device or system that has the function of analyzing the information transmitted from the image capture device and determining whether or not a person is inside the vehicle.

[0857] The "audio warning means" refers to a system that issues an audio alert to people left in the vehicle based on the detection results.

[0858] The "means for transmitting to an external communication device" is a system for transmitting alert information to a mobile phone or an internet-connected device outside the vehicle.

[0859] The "means for sending notifications to multiple registered devices" refers to a device or system that sends alert information in the form of a push notification to specific smartphones or mobile information devices that have been registered in advance.

[0860] "Means of using generative AI models to question the situation and instruct appropriate responses" refers to an AI system that uses natural language processing technology to ask questions of people in the vehicle and instruct them on what to do.

[0861] A "portable information terminal for monitoring the temperature inside a vehicle" is a portable device equipped with a temperature sensor for periodically measuring the temperature inside the vehicle and detecting any abnormalities.

[0862] The "means for detecting abnormal temperatures" is a system that analyzes temperature data and determines whether an abnormality has occurred if the temperature exceeds a set safe range.

[0863] "Means for conducting dialogue using natural language processing technology" refers to a system that uses AI technology to simulate natural conversations with people and grasp the situation.

[0864] "Means for instructing appropriate rescue methods" is a system that provides users with appropriate actions and measures based on collected information.

[0865] The "means for transmitting warning information to an external communication device" is a system for transmitting detected abnormalities or dangerous situations to an external device via a communication network.

[0866] The "means for transmitting situation information to an external communication device" is a system that transfers situation information collected through dialogue to an external device.

[0867] The system of the present invention combines a plurality of devices and technologies to operate systematically in order to prevent accidents in which kindergarten children or small children are left behind in vehicles. Embodiments of the present invention will be described in detail below.

[0868] Motion sensor camera operation

[0869] In this system, a terminal (an image capture device with a human sensor) is installed inside the vehicle. This device is equipped with a sensor and a camera to detect the presence of people inside the vehicle, and periodically collects sensor data. For example, the camera sensor detects whether there is a person inside the vehicle every 10 seconds and stores the detection results in internal memory.

[0870] By monitoring the engine's on / off status, the device detects when the vehicle's engine is turned off and the key is removed. At this time, it checks the latest sensor data and issues an audio warning if anyone remains in the vehicle.

[0871] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends push notifications to multiple pre-registered smartphones and mobile information devices. For example, a warning message such as "There is someone remaining in the vehicle!" is sent.

[0872] Operation of a portable device for monitoring temperature inside a vehicle

[0873] The terminal (personal digital assistant) is used to monitor the temperature inside the vehicle. This terminal is equipped with a temperature sensor and measures the temperature inside the vehicle every minute. If the measured temperature exceeds a set safe range (e.g., 35 degrees), the terminal detects the abnormal temperature and issues an audio warning.

[0874] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends a push notification to multiple pre-registered smartphones and mobile information devices. For example, a message such as "The temperature inside the vehicle is dangerous. Please check immediately!" may be sent.

[0875] Dialogue system using natural language processing technology

[0876] When the server receives the alert information sent from the device, it launches a generative AI model (e.g., ChatGPT) and activates a dialogue system, which asks questions to the kindergartener or toddler to confirm the situation.

[0877] Specifically, the AI ​​chatbot will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," the AI ​​will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," the AI ​​will give instructions such as, "Try to find the door lock button," supporting the child to escape on their own.

[0878] The contents of these conversations are recorded on a server and transferred in real time to an external communication device (smartphone), allowing parents and other relevant parties to immediately understand the situation and take appropriate action.

[0879] Prompt Sentence Examples

[0880] When an alert is triggered, the following prompt is entered into the generative AI model:

[0881] "We have detected that there are children remaining in the vehicle. Please begin a dialogue with the children to confirm their situation."

[0882] This enables the AI ​​model to generate appropriate dialogue depending on the situation, supporting young children and kindergarteners.

[0883] As described above, the system of the present invention can ensure the safety of kindergarten children and young children inside vehicles by combining a camera with a human sensor, a temperature monitoring mobile terminal, and a dialogue system using natural language processing technology.

[0884] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0885] Step 1:

[0886] The terminal (an image capture device with a motion sensor) collects data from the motion sensor every 10 seconds to detect the presence of people inside the vehicle.

[0887] Specific operation: When the sensor detects the presence of a person, the data (input) is stored in the camera's internal memory. The stored data is output.

[0888] Step 2:

[0889] The terminal monitors the vehicle's engine status and detects when the engine is turned off and the key is removed.

[0890] Specific operation: The system monitors the engine on / off status and key removal timing (input), and makes an alert decision based on that information. The timing of key removal and engine status are output.

[0891] Step 3:

[0892] When the key is removed, the terminal checks the latest motion sensor data to determine whether anyone remains in the vehicle.

[0893] Specific operation: Analyzes the latest stored sensor data (input), and if a person is detected, generates and issues an audio alert. The result of whether or not an alert was issued is output.

[0894] Step 4:

[0895] The terminal transmits the issued alert information to the server.

[0896] Specific operation: Alert information (input) is processed to be sent to an external communication device, and then sent to a server via a network. The server outputs a message indicating that it has received the alert information.

[0897] Step 5:

[0898] The server sends a notification to the registered terminals based on the received alert information.

[0899] Specific operation: The server analyzes the received alert data (input) and sends a push notification to the registered smartphone or information terminal. The result of the notification being sent is output.

[0900] Step 6:

[0901] The terminal (personal digital assistant) periodically monitors the temperature inside the vehicle.

[0902] Specific operation: The temperature sensor measures the temperature inside the vehicle every minute, stores the data (input) in the internal memory, and outputs the measurement results.

[0903] Step 7:

[0904] The terminal analyzes the measurement data and detects an abnormal temperature if the temperature exceeds a set temperature threshold.

[0905] Specific operation: Analyzes the measured in-car temperature data (input) and determines that an abnormality has occurred if the temperature exceeds a preset temperature threshold (e.g., 35°C). The abnormal temperature detection result is output.

[0906] Step 8:

[0907] The device will issue an audio warning if it detects an abnormal temperature.

[0908] Specific operation: Based on the abnormal temperature detection result (input), an audio alert is generated and output.

[0909] Step 9:

[0910] The terminal transmits the warning information to the server.

[0911] Specific operation: The abnormal temperature warning information (input) is processed and sent to the server via the network. The server outputs the information that it has received the warning information.

[0912] Step 10:

[0913] The server sends a notification to multiple registered terminals based on the received abnormal temperature warning information.

[0914] Specific operation: The server analyzes the abnormal temperature data (input) and sends a push notification to registered devices such as smartphones. The notification result is output.

[0915] Step 11:

[0916] The server launches a generative AI model (e.g., ChatGPT) based on the received alert information.

[0917] Specific operation: A prompt sentence (input) saying "There are children remaining in the vehicle. Please check with the children" is sent to the AI ​​model, and a dialogue scenario is generated. The generated dialogue scenario is output.

[0918] Step 12:

[0919] The server then has the generative AI model begin asking questions to the children based on the dialogue scenario.

[0920] Specific operation: Based on the prompt sentence (input), the AI ​​starts a dialogue saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" The progress of the dialogue is output.

[0921] Step 13:

[0922] The server records information obtained through the interaction and sends notifications to external communication devices.

[0923] Specific operation: The situation information (input) collected through the dialogue is stored in a database and transferred to a smartphone in real time. The notification result is output.

[0924] As described above, the various terminals and servers of this system work in cooperation to provide a quick and effective response to protect kindergarten children and young children inside vehicles.

[0925] (Application example 1)

[0926] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0927] Currently, safety management in factories places a heavy burden on workers. It is particularly difficult to grasp and respond to situations when working in high temperatures or approaching dangerous areas, making it necessary to ensure safety quickly and accurately. Furthermore, existing systems do not adequately check the detailed situation when an alert is issued, potentially threatening worker safety. Therefore, a comprehensive system is needed that can efficiently monitor temperatures in factories, detect movement using human sensors, and check the situation and issue instructions in the event of an abnormality.

[0928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0929] In this invention, the server includes: means for receiving data obtained from a motion sensor camera and detecting the presence of people within the area; means for issuing an audio alert if certain conditions are met and a person remains; means for transmitting the alert information to an external communication device; means for sending notifications to multiple registered terminals based on the transmitted alert information; means for conducting a dialogue based on the alert information using natural language processing technology; and means for gathering information on the situation through the dialogue and issuing appropriate instructions. This enables prompt and accurate safety management within the factory. Temperature changes and human activity can be monitored, and specific dialogue can be provided to ensure worker safety in the event of an abnormality.

[0930] A "camera with a human sensor" is a device that detects human movement and collects that information through the camera.

[0931] "Means for detecting the presence of people within an area" refers to technology that uses sensors or cameras to confirm that people are present within a specific area.

[0932] "A means for issuing an audio alert" refers to a device or system that issues an audio warning when an abnormality or emergency is detected.

[0933] "Means for transmitting to an external communication device" refers to technology for transmitting collected data and alert information to other external devices via the Internet or wireless communication.

[0934] The "means for sending notifications to multiple registered terminals" is a system for simultaneously sending notifications to multiple devices, such as smartphones and tablets, that have been registered in advance.

[0935] "Means of conducting dialogue using natural language processing technology" refers to technology that uses AI and machine learning to understand human language and conduct dialogue.

[0936] "Means of gathering information about the situation and issuing appropriate instructions" refers to technology that uses dialogue and sensor data to grasp the situation on-site and issue necessary instructions to people.

[0937] The "means for receiving temperature data and detecting abnormal temperatures" is a technology for receiving data from a temperature sensor and detecting that the data is outside a safe range.

[0938] "Means for generating prompt sentences using a generative AI model" refers to a technology that uses an AI learning model to automatically generate appropriate dialogue sentences or instructions based on specified content.

[0939] System configuration

[0940] This invention is a system that includes a motion sensor camera, a portable temperature monitoring device, a dialogue system using natural language processing technology, and a generative AI model. This system can perform fast and accurate monitoring to ensure the safety of workers in factories.

[0941] Hardware and Software

[0942] Hardware:

[0943] Smart glasses (e.g., Google Glass): worn by workers to display alert notifications and interactions.

[0944] Server (e.g. AWS EC2 instance): Collects data, processes it, and sends notifications.

[0945] Temperature sensor (e.g. DHT22): Monitors the temperature inside the factory.

[0946] Camera with motion sensor (e.g. Raspberry Pi Camera Module): Monitors activity within the factory.

[0947] software:

[0948] Natural language processing technology (e.g. ChatGPT): Used to communicate with workers and check the status.

[0949] Generative AI model: Used to generate prompts.

[0950] Data processing and calculation

[0951] The server receives data from cameras equipped with motion sensors and detects the presence of people within the area. Based on this information, if a person meets certain conditions (for example, if they approach a dangerous area), it issues an audio alert. It also continuously receives data from temperature sensors and detects abnormal temperatures when they exceed a set threshold. It then transmits this alert information to an external communications device, which then sends notifications to multiple registered devices.

[0952] Furthermore, a generative AI model is used to generate prompts based on the alert information, and natural language processing technology is used to conduct dialogue, confirming the worker's status in real time and issuing necessary instructions. For example, when a voice alert occurs, the server can use ChatGPT to generate a prompt such as "Please tell us your current situation" and begin dialogue.

[0953] Specific examples

[0954] If a worker working in a factory machine room experiences a sudden temperature rise and reaches 40 degrees, the smart glasses will issue a warning and send a push notification saying, "The temperature in the factory is at a dangerous level. Please evacuate!" At the same time, a dialogue system using natural language processing will ask the worker, "What is the current situation?", encouraging safe behavior. In this way, the system can monitor temperature changes and human activity, and ensure worker safety by providing specific dialogue in the event of an abnormality.

[0955] Example prompt sentence:

[0956] "The temperature in the plant is reaching dangerous levels. Please issue evacuation orders and respond."

[0957] "The motion sensor has detected a worker in a dangerous area. What is the current situation?"

[0958] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0959] Step 1:

[0960] The device (a camera with a motion sensor) periodically collects data and detects the presence of people in the area. Using the video data acquired by the camera as input, it detects human movement through an image recognition algorithm and generates human presence information as output.

[0961] Step 2:

[0962] The terminal (temperature sensor) monitors temperature data at regular intervals. It receives the temperature data measured by the sensor as input, processes it to measure the current temperature, and obtains the temperature at that time as output.

[0963] Step 3:

[0964] The server collects the data obtained from both devices and monitors the presence and temperature of people. It receives the presence and temperature data as input, stores them in a database, and prepares the processing results. As output, it obtains comprehensive data showing the current situation.

[0965] Step 4:

[0966] The server evaluates the presence and temperature data and issues an audio alert if the threshold is exceeded. It uses the monitoring data as input and triggers an audio alert if there is an anomaly. As output, it generates an audio alert and a warning message.

[0967] Step 5:

[0968] The server transmits the generated alert information to an external communication device. Using the alert information as input, the server transmits this information to the external device via the Internet or wireless communication, and as output, a warning message is distributed to multiple registered terminals.

[0969] Step 6:

[0970] The server generates a prompt sentence using a generative AI model based on the alert information. The server uses the alert information as input and generates a dialogue sentence based on the AI ​​model. The dialogue sentence is obtained as output.

[0971] Step 7:

[0972] The server uses natural language processing technology to interact with the generated prompt text. It receives the prompt text and the user's response as input and processes the dialogue. As output, it obtains the dialogue results to confirm the situation and issues instructions in real time.

[0973] Step 8:

[0974] The server stores the situation information obtained from the dialogue in a database and sends appropriate instructions to the external communication device.The server uses the dialogue results as input to generate the necessary instructions, and as output, the instructions are sent to multiple registered terminals.

[0975] Step 9:

[0976] Users take safe actions according to the notifications and alerts they receive from their devices. As input, they receive messages displayed on their devices and audio alerts, and take appropriate action. As output, worker safety is ensured.

[0977] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0978] The system of the present invention is a combination of a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine to prevent accidents involving kindergarten children and small children being left behind in vehicles. The following describes in detail the embodiments of the present invention.

[0979] Motion sensor camera operation

[0980] The terminal (camera) is equipped with a motion sensor and installed inside the vehicle. This camera periodically collects sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the motion sensor data is checked and an audio alert is issued if anyone remains inside the vehicle. The alert information is sent to a server, and notifications are sent to multiple registered terminals.

[0981] Specific examples

[0982] When the vehicle's engine is turned off and the driver removes the key, the device (camera) detects this action. If a child remains in the vehicle, the motion sensor detects their presence and issues an audio alert. The alert information is then sent to the server, and a push notification stating "There is a child remaining in the vehicle!" is sent to multiple registered smartphones.

[0983] Operation of a portable device for monitoring temperature inside a vehicle

[0984] The terminal (mobile) is a device for monitoring abnormal temperatures inside the vehicle. This mobile device periodically monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server and notified to multiple registered terminals.

[0985] Specific examples

[0986] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and issue an audio alert if it exceeds a set threshold (e.g., 35 degrees).The alert information is then sent to a server, which then sends a notification to multiple smartphones saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[0987] Dialogue system and emotion engine using natural language processing technology

[0988] The server that receives the alert information activates a chatbot system using natural language processing technology. This system asks questions to the children and young children to check their situation. This invention also incorporates an emotion engine that analyzes the children's emotional state and adjusts the dialogue accordingly.

[0989] Specific examples

[0990] When an alert is issued, ChatGPT speaks to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" When the child responds, the emotion engine analyzes the response, and if the child appears anxious, it sends a reassuring message such as "It's okay, we're calling for help." If the child appears excited, it gives calming instructions such as "Try taking a slow, deep breath."

[0991] The chatbot asks additional questions based on the child's responses and emotional state, gradually understanding the situation. For example, it asks questions such as "Where are you now?" and "Are you locked in the car?" to confirm the situation. It gives instructions on how to escape or call for help, and gives specific instructions such as "Try to find the door lock button." The content of this conversation and emotional data are sent to a server, and are then sent to multiple registered smartphones as additional information.

[0992] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine.

[0993] The processing flow will be explained below.

[0994] Processing for cameras with motion sensors

[0995] Step 1:

[0996] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[0997] Step 2:

[0998] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[0999] Step 3:

[1000] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[1001] Step 4:

[1002] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[1003] Step 5:

[1004] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[1005] Processing a portable device that monitors the temperature inside a vehicle

[1006] Step 6:

[1007] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[1008] Step 7:

[1009] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[1010] Step 8:

[1011] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[1012] Step 9:

[1013] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[1014] Step 10:

[1015] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[1016] Dialogue processing using natural language processing technology and emotion engines

[1017] Step 11:

[1018] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[1019] Step 12:

[1020] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[1021] Step 13:

[1022] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[1023] Step 14:

[1024] The chatbot uses the child's responses and analysis data from the emotion engine to determine the child's emotional state. If the child is anxious, it sends a reassuring message, such as "It's okay, we're calling for help."

[1025] Step 15:

[1026] If a child becomes agitated, the chatbot will give them calming instructions such as, "Take slow, deep breaths and stay calm."

[1027] Step 16:

[1028] The chatbot gathers information about the situation and gives instructions on how to escape or call for help, such as sending specific instructions to the child, such as "Try to find the door lock button."

[1029] Step 17:

[1030] The chatbot transmits the content of the conversation and emotional data to the server.

[1031] Step 18:

[1032] The server then sends additional push notifications to the registered smartphone ID based on the collected situational information and emotion data, such as a notification that "A child is locked in a car. Please check immediately."

[1033] Through the above steps, the system of the present invention can ensure the safety of kindergarteners and young children through the collaboration of a camera with a motion sensor, a portable temperature monitoring device, natural language processing technology, and an emotion engine.

[1034] Example 2

[1035] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1036] The problem to be solved by this invention is to prevent accidents involving kindergarten children and small children being left behind in vehicles and ensure their safety. In particular, the object is to provide a system that quickly and accurately detects abnormalities and provides necessary notifications and dialogue when the vehicle engine is stopped and the key is removed, or when the temperature inside the vehicle reaches a dangerous level.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1038] In this invention, the server includes: means for receiving data from a motion sensor camera to detect the presence of a person inside the vehicle; means for issuing an audio alert if a person remains inside the vehicle when the vehicle engine is stopped and the key is removed; means for transmitting the alert information to a remote computer; means for transmitting notifications to multiple communication devices based on the transmitted alert information; means for receiving temperature data from a portable device that monitors the temperature inside the vehicle and detecting an abnormal temperature; means for issuing an audio alert when an abnormal temperature is detected; means for transmitting the alert information to the remote computer; means for conducting a dialogue using natural language processing technology; means for gathering information about the situation through the dialogue and providing instructions on how to call for help; and means for transmitting the information about the situation to the remote computer. This allows the server to quickly detect the presence of a child or toddler inside the vehicle and an abnormal temperature, and to respond immediately by transmitting a remote notification along with an audio alert. Furthermore, the server can use natural language processing technology to dialogue with the child to accurately understand the situation and reassure them.

[1039] A "camera with a human sensor" is a photographic device installed inside a vehicle and equipped with a sensor to detect human movement and presence.

[1040] The "means for receiving data" refers to a device or system for acquiring information transmitted from a camera with a motion sensor or a mobile device.

[1041] "Means for issuing an audio alert" refers to a function or device for issuing an audio warning when an abnormality is detected inside the vehicle.

[1042] The "means for transmitting alert information to an external communication device" refers to a function or system for transferring detected abnormality information to an external server or communication device using the Internet or other communication means.

[1043] The "means for sending notifications" is a system for sending warnings and notifications to multiple registered terminals based on alert information sent to an external communication device.

[1044] A "portable temperature monitoring device" is a portable device used to measure and monitor the temperature inside a vehicle.

[1045] "Means for detecting abnormal temperatures" refers to functions or systems that analyze data obtained from temperature monitoring equipment and detect abnormal temperatures that exceed a set threshold.

[1046] "Means for conducting dialogue using natural language processing technology" refers to a system that uses artificial intelligence technology to understand human language and conduct dialogue.

[1047] "Means for gathering information about the situation and providing instructions on how to seek help" is a system that understands the user's situation through dialogue and provides instructions on appropriate measures and actions.

[1048] The "means for transmitting status information to an external communication device" refers to a function or system for transferring information collected through dialogue to an external communication device or server.

[1049] As an embodiment of the present invention, a system is provided that combines various sensors and communication means to prevent accidents involving the abandonment of kindergarten children or small children in vehicles. The following describes in detail each component of this system and its operation.

[1050] Motion sensor camera operation

[1051] The device (camera with motion sensor) is installed inside the vehicle and is equipped with a sensor that detects the presence of people. Specifically, a Raspberry Pi is used, to which a motion sensor is attached. The camera periodically collects sensor data and detects the movement of people inside the vehicle. When the vehicle engine is turned off and the key is removed, the device checks the motion sensor data and issues an audio alert if anyone remains inside the vehicle. The alert information is sent to a remote computer such as AWS Lambda, which then uses Firebase Cloud Messaging to send notifications to multiple registered communication devices.

[1052] Specific examples

[1053] For example, if the vehicle engine is turned off and the driver removes the key, the device (camera) will detect this action. If a child remains in the vehicle, the human sensor will detect their presence and issue a voice alert saying, "Caution! There is someone left in the vehicle!" The alert information is then sent to the server, and a push notification saying, "Emergency! There is a child left in the vehicle!" is sent to multiple registered smartphones.

[1054] Operation of a portable device for monitoring temperature inside a vehicle

[1055] The terminal (portable device) is used to monitor the temperature inside the vehicle. Specifically, it uses an Arduino with a temperature sensor attached. The portable device periodically measures the temperature inside the vehicle and issues an audio alert if it exceeds a threshold preset by the user. The alert information is sent to a server, which then sends notifications to multiple communication devices.

[1056] Specific examples

[1057] For example, if the temperature inside a vehicle reaches 40 degrees on a hot summer day, the terminal (mobile device) detects this and issues a voice alert saying, "The temperature inside the vehicle is very high!" The alert information is then sent to the server, which then sends a notification to multiple registered smartphones saying, "Temperature warning! The temperature inside the vehicle is dangerous, please check immediately!"

[1058] Dialogue system and emotion engine operation using natural language processing technology

[1059] The server that receives the alert information uses ChatGPT to launch a dialogue system using natural language processing technology. This system asks questions to the children and toddlers to confirm their situation. This invention also incorporates an emotion engine such as Affectiva to analyze the children's emotional state and adjust the dialogue accordingly.

[1060] Specific examples

[1061] For example, when an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. Can you hear me, XX-chan?" If the child responds with "Help me!", the emotion engine will analyze the response and determine that the child is in an anxious state. The dialogue system will then send a message saying, "It's okay, I'm calling for help. Can you tell me where you are?" This information is stored on the server and sent to the parent's smartphone as additional information.

[1062] Prompt Sentence Examples

[1063] "If a child remains in the car, how do you speak to them, check in on them, and provide them with the support they need?"

[1064] In this way, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine. Each component is realized using specific hardware and software, and the results are notified to parents and guardians in real time, allowing for quick and appropriate response.

[1065] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1066] Processing for cameras with motion sensors

[1067] Step 1:

[1068] The device (camera) periodically collects data from the human presence sensor. In this case, the input is data on the presence and movement of people detected by the sensor. Specifically, a Raspberry Pi is used to acquire data from the sensor at regular intervals. The output is information on whether a person has been detected.

[1069] Step 2:

[1070] The terminal (camera) detects when the vehicle engine is stopped and the key is removed. The input is engine status information obtained from the vehicle's internal system. The output is a signal indicating that the engine has been stopped and the key has been removed. For example, the terminal can monitor the engine status in cooperation with the vehicle's system and recognize that the key has been removed.

[1071] Step 3:

[1072] The terminal (camera) checks the motion sensor data when the engine is stopped and determines whether or not there is anyone left in the vehicle. The input is the motion sensor data acquired in step 1 and the engine stop information in step 2. The output is a determination result of whether or not there is anyone left in the vehicle. Specifically, the sensor data is analyzed, and if there is anyone left, the system proceeds to the next operation.

[1073] Step 4:

[1074] If the device (camera) determines that a person remains in the vehicle, it issues a voice alert through the speaker. The input is the judgment result obtained in step 3. The output is an audio alert. For example, it may issue a message saying, "Caution! There is a person remaining in the vehicle!"

[1075] Step 5:

[1076] The device (camera) issues an audio alert and simultaneously sends the alert information to the server. The input is the audio alert information from step 4. The output is the alert information that is sent to a remote computer. Specifically, the alert information is sent to the cloud using AWS Lambda.

[1077] Step 6:

[1078] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message saying "Emergency! A child is left in the car!" is sent to the smartphone via Firebase Cloud Messaging.

[1079] Processing a portable device that monitors the temperature inside a vehicle

[1080] Step 1:

[1081] The terminal (mobile device) periodically measures the temperature inside the car. The input is data from the temperature sensor. Specifically, the temperature is periodically acquired using Arduino. The output is the current temperature information inside the car.

[1082] Step 2:

[1083] The terminal (mobile device) checks the temperature threshold set by the user in advance. The input is the threshold information set in the application. The output is the set threshold itself. Specifically, the threshold is set to 35 degrees in the application.

[1084] Step 3:

[1085] The terminal (portable device) compares the temperature data with a threshold value and detects abnormal temperatures. The inputs are the temperature information from step 1 and the threshold value information from step 2. The output is a determination result as to whether the temperature has exceeded the threshold value. Specifically, if the temperature inside the vehicle reaches 40 degrees, it is determined to be abnormal.

[1086] Step 4:

[1087] If the terminal (mobile device) detects an abnormal temperature, it issues a voice alert. The input is the judgment result of step 3. The output is an audio alert. For example, it may issue a warning saying, "The temperature inside the car is very high!"

[1088] Step 5:

[1089] The terminal (mobile device) issues a voice alert and simultaneously transmits the alert information to the server. The input is the alert information from step 4. The output is the alert information to be sent to a remote computer. Specifically, the terminal transmits information about abnormal temperature detection to the server.

[1090] Step 6:

[1091] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message is sent to the smartphone saying, "Temperature warning! The temperature inside the car is dangerous, please check immediately!"

[1092] Dialogue systems and emotion engines using natural language processing technology

[1093] Step 1:

[1094] The server launches a dialogue system using natural language processing technology based on the alert information received from the device. The input is the alert information. The output is that the dialogue system is ready to launch. Specifically, it launches a chatbot using ChatGPT.

[1095] Step 2:

[1096] The dialogue system launched by the server speaks to the kindergartener or toddler. The input is the initial question to the kindergartener. The output is a message to the kindergartener saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" Specifically, the system generates a message to gently inquire about the situation for the child.

[1097] Step 3:

[1098] The server receives the child's response and analyzes their emotional state using an emotion engine. The input is the child's response. The output is the analysis result of the child's emotional state. Specifically, the response "Help me!" is received and analyzed as an anxious state.

[1099] Step 4:

[1100] The server adjusts the dialogue content according to the emotional state. The input is the analysis result of the emotion engine. The output is an appropriately adjusted dialogue message. Specifically, it sends the message, "I'm OK, I'm calling for help. Can you tell me where you are?"

[1101] Step 5:

[1102] The server then provides more appropriate instructions based on the situational information collected through the dialogue. The input is the child's location and situation. The output is additional, more specific instructions. For example, the server might give instructions such as, "Try to find the door lock button."

[1103] Step 6:

[1104] The server notifies the registered communication devices of the content of the conversation and emotional data. The input is the content of the conversation and the emotional data. The output is a push notification with additional information. Specifically, a message is sent to the parent's smartphone saying, "Your child is asking for help. Please tell them to find the door lock button."

[1105] Through the above specific processing steps, the system of the present invention realizes a quick and accurate response to ensure the safety of kindergarten children and young children.

[1106] (Application example 2)

[1107] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] In recent years, as self-driving vehicles have become more common, accidents involving kindergarten children and young children being left behind inside the vehicle have become a problem. Such accidents are particularly likely to occur when the driver leaves the vehicle, posing a serious risk to the children's lives. Furthermore, the temperature inside the vehicle may rise, putting children at risk of heatstroke. Furthermore, if a child panics inside the vehicle, it becomes difficult to respond appropriately. For these reasons, a comprehensive system to ensure the safety of kindergarten children and young children is needed in self-driving vehicles.

[1109] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving data obtained from a motion sensor camera and detecting the presence of a person in the vehicle, means for issuing an audio alert if a person remains in the vehicle when the vehicle key is removed, means for transmitting the alert information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted alert information, means for conducting a dialogue using natural language processing technology, means for collecting information about the situation through the dialogue, means for analyzing responses using an emotion analysis engine and generating an appropriate message based on the responses, and means for instructing how to call for help based on the dialogue and analyzed information. This makes it possible to quickly and accurately grasp the situation of a kindergartener or young child left behind in a vehicle and safely deal with the situation.

[1110] A "camera with a human sensor" is a camera that detects the presence of people inside a vehicle and is a device that has a built-in human sensor.

[1111] A "vehicle key" is a physical or electronic key used to start or stop a vehicle engine.

[1112] "Voice alert" is a function that issues a voice warning if an abnormality is detected inside the vehicle.

[1113] The "external communication device" is a communication device installed outside the vehicle, which transmits and receives information via the Internet.

[1114] "Multiple terminals" refers to terminal devices connected to a communication network, such as smartphones and tablets owned by registered users.

[1115] "Natural language processing technology" is a technology for understanding human language and engaging in dialogue, and is a technology for analyzing and generating text.

[1116] An "emotion analysis engine" is a program that analyzes a user's emotional state based on data such as their statements and facial expressions.

[1117] "Dialogue means" refers to a means for communicating with a user, and is an interactive interface that uses voice or text.

[1118] "Instructions on how to call for help" refers to a function that provides specific instructions that children and young children need to call for help while in a vehicle.

[1119] MODE FOR CARRYING OUT THE INVENTION

[1120] The system of the present invention is designed to prevent accidents involving the abandonment of kindergarten children and young children in self-driving vehicles. The system consists of the following main components: a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, an emotion analysis engine, and a server-based system.

[1121] System Configuration

[1122] 1. Motion Sensor Camera:

[1123] The terminal, a motion sensor camera, is installed inside the vehicle and periodically collects sensor data. This camera has the function of detecting whether there is a person inside the vehicle and confirming the presence of a person when the car key is removed. If there is a person inside, it issues an audio alert and sends the corresponding alert information to the server.

[1124] 2. Portable temperature monitoring device:

[1125] The terminal, a portable temperature monitoring device, monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. This information is also sent to a server, which then sends notifications to multiple terminals.

[1126] 3. Dialogue systems using natural language processing technology:

[1127] The server activates a chatbot using natural language processing technology to converse with the children in the car. The system asks questions to check the children's status and analyzes their responses using an emotion analysis engine.

[1128] 4. Sentiment Analysis Engine:

[1129] The emotion analysis engine analyzes the emotional state of the children during the conversation and guides them to an appropriate response accordingly, a process that is important for reducing the anxiety and panic that the children may be feeling.

[1130] 5. Server and Notification System:

[1131] The server aggregates alert information and the content of conversations between the children and the nursery school, and sends notifications to multiple devices via an external communication device, allowing relevant parties to quickly grasp the situation and take appropriate action.

[1132] The specific hardware and software used

[1133] Camera and motion sensor: Using development kits such as Arduino, a dedicated camera is installed inside the vehicle.

[1134] Temperature sensor: Connect a temperature sensor to the Raspberry Pi to obtain data in real time.

[1135] Server: Uses cloud services such as AWS and Google Cloud Platform.

[1136] Natural language processing technology: A dialogue system using OpenAI's GPT-3.5.

[1137] Sentiment analysis engine: Uses Microsoft Azure's emotion recognition API, etc.

[1138] Specific examples

[1139] For example, suppose a vehicle's engine shuts off on a hot summer day, and the driver leaves the vehicle and removes the key. If a child is left inside the vehicle, a motion sensor camera will detect the child's presence and issue an audio alert. At the same time, a portable temperature monitoring device will detect that the temperature inside the vehicle has exceeded 35 degrees and also issue an audio alert. This alert information is sent to a server, and push notifications are sent to multiple registered smartphones, stating, "There is a child left inside the vehicle!" and "The temperature inside the vehicle is dangerous, please check immediately!"

[1140] The server then launches a dialogue system using natural language processing technology and begins asking questions to the child, such as "Hello, Mr. / Ms. X, can you hear me?" If the child responds, the emotion analysis engine analyzes the response and, if the child seems anxious, offers reassuring words like "It's okay, I'm calling for help."

[1141] Prompt Sentence Examples

[1142] Child: "Yes, I can hear you."

[1143] Dialogue system: "Hello, Mr. / Ms. X. We've detected an in-car alert. Where are you now?"

[1144] Kindergartener: "I'm near the door."

[1145] Dialogue system: "Don't worry. Someone will come to help you shortly. Try finding the door lock button."

[1146] In this way, the system of the present invention can take comprehensive measures to ensure child safety.

[1147] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1148] Step 1:

[1149] The terminal collects data inside the vehicle using a camera with a motion sensor and a portable temperature monitoring device. The motion sensor detects whether a person is inside the vehicle, and the portable temperature monitoring device monitors the temperature inside the vehicle. The data obtained from these sensors (input) is sent to the terminal as human presence and temperature information (output).

[1150] Step 2:

[1151] The terminal processes the collected data and makes a decision based on the decision criteria. Specifically, if the motion sensor detects a person inside the vehicle, or if the temperature sensor exceeds a set threshold (e.g., 35 degrees), an audio alert is issued. The terminal then issues an audio alert and sends the information to the server (input data: presence of a person, temperature information; output data: alert information).

[1152] Step 3:

[1153] The server receives alert information sent from the device. The received alert information is about the presence of people and abnormal temperatures (input). Based on this information, the server sends push notifications to multiple registered devices (output). At this stage, the alert information is delivered to the user's smartphone or tablet.

[1154] Step 4:

[1155] Upon receiving the alert information, the server activates a dialogue system using natural language processing technology. This dialogue system uses a generative AI model to generate appropriate dialogue prompts for kindergarteners and young children (input data: alert information; output data: dialogue prompts).

[1156] Step 5:

[1157] The user (a child or toddler) responds to prompts from the dialogue system. The user's response (input) is received in real time by the dialogue system. The server analyzes this response using a generative AI model and generates an appropriate reply (output).

[1158] Step 6:

[1159] The server uses an emotion analysis engine to analyze the user's emotional state from their response. Based on the analysis results, the server adjusts the content of the dialogue (input data: user's response; output data: emotion analysis results). For example, if the user shows signs of anxiety, it generates a reassuring message.

[1160] Step 7:

[1161] The server takes into account the dialogue and the results of sentiment analysis to generate specific instructions for asking for help. It generates prompts for appropriate actions along with the dialogue content and provides them to the user (input data: dialogue history, sentiment analysis results; output data: specific instructions). For example, it may give specific instructions such as "Try to find the door lock button."

[1162] Step 8:

[1163] The server sends dialogue and emotion analysis data to an external communication device and notifies multiple registered devices of the situation (input data: dialogue history, emotion analysis results; output data: situation notification). This allows the relevant parties to grasp the situation in real time.

[1164] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1165] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1166] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1167] [Fourth embodiment]

[1168] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1169] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1170] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1171] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1172] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1173] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1174] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1175] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1176] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1177] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1178] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1179] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1180] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1181] The system of the present invention provides a means for preventing accidents in which kindergarten children or small children are left behind in vehicles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[1182] Motion sensor camera operation

[1183] The terminal (camera) is installed inside a vehicle equipped with a motion sensor. This camera periodically collects motion sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the camera checks the latest sensor data at that time and issues an audio alert if someone remains inside. In addition, by sending the alert information to a server, a notification is sent to an external communication device.

[1184] Specific examples

[1185] When the vehicle's engine is shut off and the driver removes the key, the device (camera) detects this action. At the same time, if there are any children remaining in the vehicle, the motion sensor detects their presence and issues an audio alert. In addition, the alert information is sent to the server, and a push notification saying "There are children remaining in the vehicle!" is sent to multiple registered smartphones.

[1186] Operation of a portable device for monitoring temperature inside a vehicle

[1187] The terminal (mobile phone) is used to monitor abnormal temperatures inside the vehicle. This device monitors the temperature inside the vehicle at regular intervals and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server, which then sends a notification to an external communication device.

[1188] Specific examples

[1189] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and determine that the temperature is abnormal if it exceeds 35 degrees. An audio alert will be issued and the alert information will be sent to the server. Based on this information, the server will send a push notification to the registered smartphone saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[1190] Dialogue system using natural language processing technology

[1191] The server that receives the alert information activates a dialogue system (such as an AI chatbot) that uses natural language processing technology. This system asks questions to the kindergarten or toddler to confirm the situation. Based on the information obtained through this dialogue, it gives instructions on how to call for help or escape appropriately.

[1192] Specific examples

[1193] When an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," ChatGPT will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," ChatGPT will give instructions such as, "Try to find the door lock button," helping the child escape on their own. The content of this conversation is sent to the server and notified to the registered smartphone as additional information.

[1194] As a result, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and an interactive system using natural language processing technology.

[1195] The processing flow will be explained below.

[1196] Processing for cameras with motion sensors

[1197] Step 1:

[1198] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[1199] Step 2:

[1200] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[1201] Step 3:

[1202] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[1203] Step 4:

[1204] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[1205] Step 5:

[1206] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[1207] Processing a portable device that monitors the temperature inside a vehicle

[1208] Step 6:

[1209] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[1210] Step 7:

[1211] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[1212] Step 8:

[1213] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[1214] Step 9:

[1215] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[1216] Step 10:

[1217] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[1218] Dialogue processing using natural language processing technology

[1219] Step 11:

[1220] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[1221] Step 12:

[1222] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[1223] Step 13:

[1224] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[1225] Step 14:

[1226] Based on the child's responses, the chatbot will give them instructions on how to escape or call for help, such as "Try to find the door lock button."

[1227] Step 15:

[1228] The chatbot transmits the situation information obtained through the dialogue to the server.

[1229] Step 16:

[1230] The server then sends additional push notifications to the registered smartphone ID based on the collected situation information, such as a notification that "A child is locked in a car. Please check the situation."

[1231] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, and a dialogue system using natural language processing technology.

[1232] Example 1

[1233] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] Accidents involving kindergarten children and young children being left behind in vehicles are a serious risk, especially in high-temperature environments. Conventional warning systems only detect the presence of people in the vehicle or abnormally high temperatures, but do not provide specific countermeasures, making immediate rescue difficult in many cases. Furthermore, warnings cannot be properly communicated to nearby adults, resulting in insufficient prevention of accidents. Therefore, there is a need for a system that can protect kindergarten children and young children by providing specific instructions and dialogue tailored to the situation.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1236] In this invention, the server includes means for receiving data obtained from an image capture device with a motion sensor and detecting the presence of a person in the vehicle, means for issuing an audio warning if a person remains in the vehicle when the vehicle engine is stopped and the key is removed, means for transmitting the warning information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted warning information, and means for using a generative AI model to inquire about the situation and instruct appropriate responses. This makes it possible to provide not only warnings but also specific escape instructions and calls for help, enabling quick responses to prevent accidents and ensure the safety of kindergarteners and young children.

[1237] An "image capture device with a human presence sensor" is a device that is installed inside a vehicle and combines a sensor and a camera to detect the presence of people.

[1238] "Means for receiving data and detecting the presence of a person inside the vehicle" refers to a device or system that has the function of analyzing the information transmitted from the image capture device and determining whether or not a person is inside the vehicle.

[1239] The "audio warning means" refers to a system that issues an audio alert to people left in the vehicle based on the detection results.

[1240] The "means for transmitting to an external communication device" is a system for transmitting alert information to a mobile phone or an internet-connected device outside the vehicle.

[1241] The "means for sending notifications to multiple registered devices" refers to a device or system that sends alert information in the form of a push notification to specific smartphones or mobile information devices that have been registered in advance.

[1242] "Means of using generative AI models to question the situation and instruct appropriate responses" refers to an AI system that uses natural language processing technology to ask questions of people in the vehicle and instruct them on what to do.

[1243] A "portable information terminal for monitoring the temperature inside a vehicle" is a portable device equipped with a temperature sensor for periodically measuring the temperature inside the vehicle and detecting any abnormalities.

[1244] The "means for detecting abnormal temperatures" is a system that analyzes temperature data and determines whether an abnormality has occurred if the temperature exceeds a set safe range.

[1245] "Means for conducting dialogue using natural language processing technology" refers to a system that uses AI technology to simulate natural conversations with people and grasp the situation.

[1246] "Means for instructing appropriate rescue methods" is a system that provides users with appropriate actions and measures based on collected information.

[1247] The "means for transmitting warning information to an external communication device" is a system for transmitting detected abnormalities or dangerous situations to an external device via a communication network.

[1248] The "means for transmitting situation information to an external communication device" is a system that transfers situation information collected through dialogue to an external device.

[1249] The system of the present invention combines a plurality of devices and technologies to operate systematically in order to prevent accidents in which kindergarten children or small children are left behind in vehicles. Embodiments of the present invention will be described in detail below.

[1250] Motion sensor camera operation

[1251] In this system, a terminal (an image capture device with a human sensor) is installed inside the vehicle. This device is equipped with a sensor and a camera to detect the presence of people inside the vehicle, and periodically collects sensor data. For example, the camera sensor detects whether there is a person inside the vehicle every 10 seconds and stores the detection results in internal memory.

[1252] By monitoring the engine's on / off status, the device detects when the vehicle's engine is turned off and the key is removed. At this time, it checks the latest sensor data and issues an audio warning if anyone remains in the vehicle.

[1253] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends push notifications to multiple pre-registered smartphones and mobile information devices. For example, a warning message such as "There is someone remaining in the vehicle!" is sent.

[1254] Operation of a portable device for monitoring temperature inside a vehicle

[1255] The terminal (personal digital assistant) is used to monitor the temperature inside the vehicle. This terminal is equipped with a temperature sensor and measures the temperature inside the vehicle every minute. If the measured temperature exceeds a set safe range (e.g., 35 degrees), the terminal detects the abnormal temperature and issues an audio warning.

[1256] At the same time as issuing the warning, the device sends the alert information to a server. Based on the received alert information, the server then sends a push notification to multiple pre-registered smartphones and mobile information devices. For example, a message such as "The temperature inside the vehicle is dangerous. Please check immediately!" may be sent.

[1257] Dialogue system using natural language processing technology

[1258] When the server receives the alert information sent from the device, it launches a generative AI model (e.g., ChatGPT) and activates a dialogue system, which asks questions to the kindergartener or toddler to confirm the situation.

[1259] Specifically, the AI ​​chatbot will speak to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" If the child replies "Yes," the AI ​​will then ask, "Where are you now?" to confirm the situation. If the child replies, "I'm in the car," the AI ​​will give instructions such as, "Try to find the door lock button," supporting the child to escape on their own.

[1260] The contents of these conversations are recorded on a server and transferred in real time to an external communication device (smartphone), allowing parents and other relevant parties to immediately understand the situation and take appropriate action.

[1261] Prompt Sentence Examples

[1262] When an alert is triggered, the following prompt is entered into the generative AI model:

[1263] "We have detected that there are children remaining in the vehicle. Please begin a dialogue with the children to confirm their situation."

[1264] This enables the AI ​​model to generate appropriate dialogue depending on the situation, supporting young children and kindergarteners.

[1265] As described above, the system of the present invention can ensure the safety of kindergarten children and young children inside vehicles by combining a camera with a human sensor, a temperature monitoring mobile terminal, and a dialogue system using natural language processing technology.

[1266] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1267] Step 1:

[1268] The terminal (an image capture device with a motion sensor) collects data from the motion sensor every 10 seconds to detect the presence of people inside the vehicle.

[1269] Specific operation: When the sensor detects the presence of a person, the data (input) is stored in the camera's internal memory. The stored data is output.

[1270] Step 2:

[1271] The terminal monitors the vehicle's engine status and detects when the engine is turned off and the key is removed.

[1272] Specific operation: The system monitors the engine on / off status and key removal timing (input), and makes an alert decision based on that information. The timing of key removal and engine status are output.

[1273] Step 3:

[1274] When the key is removed, the terminal checks the latest motion sensor data to determine whether anyone remains in the vehicle.

[1275] Specific operation: Analyzes the latest stored sensor data (input), and if a person is detected, generates and issues an audio alert. The result of whether or not an alert was issued is output.

[1276] Step 4:

[1277] The terminal transmits the issued alert information to the server.

[1278] Specific operation: Alert information (input) is processed to be sent to an external communication device, and then sent to a server via a network. The server outputs a message indicating that it has received the alert information.

[1279] Step 5:

[1280] The server sends a notification to the registered terminals based on the received alert information.

[1281] Specific operation: The server analyzes the received alert data (input) and sends a push notification to the registered smartphone or information terminal. The result of the notification being sent is output.

[1282] Step 6:

[1283] The terminal (personal digital assistant) periodically monitors the temperature inside the vehicle.

[1284] Specific operation: The temperature sensor measures the temperature inside the vehicle every minute, stores the data (input) in the internal memory, and outputs the measurement results.

[1285] Step 7:

[1286] The terminal analyzes the measurement data and detects an abnormal temperature if the temperature exceeds a set temperature threshold.

[1287] Specific operation: Analyzes the measured in-car temperature data (input) and determines that an abnormality has occurred if the temperature exceeds a preset temperature threshold (e.g., 35°C). The abnormal temperature detection result is output.

[1288] Step 8:

[1289] The device will issue an audio warning if it detects an abnormal temperature.

[1290] Specific operation: Based on the abnormal temperature detection result (input), an audio alert is generated and output.

[1291] Step 9:

[1292] The terminal transmits the warning information to the server.

[1293] Specific operation: The abnormal temperature warning information (input) is processed and sent to the server via the network. The server outputs the information that it has received the warning information.

[1294] Step 10:

[1295] The server sends a notification to multiple registered terminals based on the received abnormal temperature warning information.

[1296] Specific operation: The server analyzes the abnormal temperature data (input) and sends a push notification to registered devices such as smartphones. The notification result is output.

[1297] Step 11:

[1298] The server launches a generative AI model (e.g., ChatGPT) based on the received alert information.

[1299] Specific operation: A prompt sentence (input) saying "There are children remaining in the vehicle. Please check with the children" is sent to the AI ​​model, and a dialogue scenario is generated. The generated dialogue scenario is output.

[1300] Step 12:

[1301] The server then has the generative AI model begin asking questions to the children based on the dialogue scenario.

[1302] Specific operation: Based on the prompt sentence (input), the AI ​​starts a dialogue saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" The progress of the dialogue is output.

[1303] Step 13:

[1304] The server records information obtained through the interaction and sends notifications to external communication devices.

[1305] Specific operation: The situation information (input) collected through the dialogue is stored in a database and transferred to a smartphone in real time. The notification result is output.

[1306] As described above, the various terminals and servers of this system work in cooperation to provide a quick and effective response to protect kindergarten children and young children inside vehicles.

[1307] (Application example 1)

[1308] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] Currently, safety management in factories places a heavy burden on workers. It is particularly difficult to grasp and respond to situations when working in high temperatures or approaching dangerous areas, making it necessary to ensure safety quickly and accurately. Furthermore, existing systems do not adequately check the detailed situation when an alert is issued, potentially threatening worker safety. Therefore, a comprehensive system is needed that can efficiently monitor temperatures in factories, detect movement using human sensors, and check the situation and issue instructions in the event of an abnormality.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1311] In this invention, the server includes: means for receiving data obtained from a motion sensor camera and detecting the presence of people within the area; means for issuing an audio alert if certain conditions are met and a person remains; means for transmitting the alert information to an external communication device; means for sending notifications to multiple registered terminals based on the transmitted alert information; means for conducting a dialogue based on the alert information using natural language processing technology; and means for gathering information on the situation through the dialogue and issuing appropriate instructions. This enables prompt and accurate safety management within the factory. Temperature changes and human activity can be monitored, and specific dialogue can be provided to ensure worker safety in the event of an abnormality.

[1312] A "camera with a human sensor" is a device that detects human movement and collects that information through the camera.

[1313] "Means for detecting the presence of people within an area" refers to technology that uses sensors or cameras to confirm that people are present within a specific area.

[1314] "A means for issuing an audio alert" refers to a device or system that issues an audio warning when an abnormality or emergency is detected.

[1315] "Means for transmitting to an external communication device" refers to technology for transmitting collected data and alert information to other external devices via the Internet or wireless communication.

[1316] The "means for sending notifications to multiple registered terminals" is a system for simultaneously sending notifications to multiple devices, such as smartphones and tablets, that have been registered in advance.

[1317] "Means of conducting dialogue using natural language processing technology" refers to technology that uses AI and machine learning to understand human language and conduct dialogue.

[1318] "Means of gathering information about the situation and issuing appropriate instructions" refers to technology that uses dialogue and sensor data to grasp the situation on-site and issue necessary instructions to people.

[1319] The "means for receiving temperature data and detecting abnormal temperatures" is a technology for receiving data from a temperature sensor and detecting that the data is outside a safe range.

[1320] "Means for generating prompt sentences using a generative AI model" refers to a technology that uses an AI learning model to automatically generate appropriate dialogue sentences or instructions based on specified content.

[1321] System configuration

[1322] This invention is a system that includes a motion sensor camera, a portable temperature monitoring device, a dialogue system using natural language processing technology, and a generative AI model. This system can perform fast and accurate monitoring to ensure the safety of workers in factories.

[1323] Hardware and Software

[1324] Hardware:

[1325] Smart glasses (e.g., Google Glass): worn by workers to display alert notifications and interactions.

[1326] Server (e.g. AWS EC2 instance): Collects data, processes it, and sends notifications.

[1327] Temperature sensor (e.g. DHT22): Monitors the temperature inside the factory.

[1328] Camera with motion sensor (e.g. Raspberry Pi Camera Module): Monitors activity within the factory.

[1329] software:

[1330] Natural language processing technology (e.g. ChatGPT): Used to communicate with workers and check the status.

[1331] Generative AI model: Used to generate prompts.

[1332] Data processing and calculation

[1333] The server receives data from cameras equipped with motion sensors and detects the presence of people within the area. Based on this information, if a person meets certain conditions (for example, if they approach a dangerous area), it issues an audio alert. It also continuously receives data from temperature sensors and detects abnormal temperatures when they exceed a set threshold. It then transmits this alert information to an external communications device, which then sends notifications to multiple registered devices.

[1334] Furthermore, a generative AI model is used to generate prompts based on the alert information, and natural language processing technology is used to conduct dialogue, confirming the worker's status in real time and issuing necessary instructions. For example, when a voice alert occurs, the server can use ChatGPT to generate a prompt such as "Please tell us your current situation" and begin dialogue.

[1335] Specific examples

[1336] If a worker working in a factory machine room experiences a sudden temperature rise and reaches 40 degrees, the smart glasses will issue a warning and send a push notification saying, "The temperature in the factory is at a dangerous level. Please evacuate!" At the same time, a dialogue system using natural language processing will ask the worker, "What is the current situation?", encouraging safe behavior. In this way, the system can monitor temperature changes and human activity, and ensure worker safety by providing specific dialogue in the event of an abnormality.

[1337] Example prompt sentence:

[1338] "The temperature in the plant is reaching dangerous levels. Please issue evacuation orders and respond."

[1339] "The motion sensor has detected a worker in a dangerous area. What is the current situation?"

[1340] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1341] Step 1:

[1342] The device (a camera with a motion sensor) periodically collects data and detects the presence of people in the area. Using the video data acquired by the camera as input, it detects human movement through an image recognition algorithm and generates human presence information as output.

[1343] Step 2:

[1344] The terminal (temperature sensor) monitors temperature data at regular intervals. It receives the temperature data measured by the sensor as input, processes it to measure the current temperature, and obtains the temperature at that time as output.

[1345] Step 3:

[1346] The server collects the data obtained from both devices and monitors the presence and temperature of people. It receives the presence and temperature data as input, stores them in a database, and prepares the processing results. As output, it obtains comprehensive data showing the current situation.

[1347] Step 4:

[1348] The server evaluates the presence and temperature data and issues an audio alert if the threshold is exceeded. It uses the monitoring data as input and triggers an audio alert if there is an anomaly. As output, it generates an audio alert and a warning message.

[1349] Step 5:

[1350] The server transmits the generated alert information to an external communication device. Using the alert information as input, the server transmits this information to the external device via the Internet or wireless communication, and as output, a warning message is distributed to multiple registered terminals.

[1351] Step 6:

[1352] The server generates a prompt sentence using a generative AI model based on the alert information. The server uses the alert information as input and generates a dialogue sentence based on the AI ​​model. The dialogue sentence is obtained as output.

[1353] Step 7:

[1354] The server uses natural language processing technology to interact with the generated prompt text. It receives the prompt text and the user's response as input and processes the dialogue. As output, it obtains the dialogue results to confirm the situation and issues instructions in real time.

[1355] Step 8:

[1356] The server stores the situation information obtained from the dialogue in a database and sends appropriate instructions to the external communication device.The server uses the dialogue results as input to generate the necessary instructions, and as output, the instructions are sent to multiple registered terminals.

[1357] Step 9:

[1358] Users take safe actions according to the notifications and alerts they receive from their devices. As input, they receive messages displayed on their devices and audio alerts, and take appropriate action. As output, worker safety is ensured.

[1359] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1360] The system of the present invention is a combination of a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine to prevent accidents involving kindergarten children and small children being left behind in vehicles. The following describes in detail the embodiments of the present invention.

[1361] Motion sensor camera operation

[1362] The terminal (camera) is equipped with a motion sensor and installed inside the vehicle. This camera periodically collects sensor data and detects the presence of people inside the vehicle. When the vehicle key is removed, the motion sensor data is checked and an audio alert is issued if anyone remains inside the vehicle. The alert information is sent to a server, and notifications are sent to multiple registered terminals.

[1363] Specific examples

[1364] When the vehicle's engine is turned off and the driver removes the key, the device (camera) detects this action. If a child remains in the vehicle, the motion sensor detects their presence and issues an audio alert. The alert information is then sent to the server, and a push notification stating "There is a child remaining in the vehicle!" is sent to multiple registered smartphones.

[1365] Operation of a portable device for monitoring temperature inside a vehicle

[1366] The terminal (mobile) is a device for monitoring abnormal temperatures inside the vehicle. This mobile device periodically monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. The alert information is sent to a server and notified to multiple registered terminals.

[1367] Specific examples

[1368] If the temperature inside the vehicle rises suddenly on a hot summer day, the device (mobile phone) will detect this and issue an audio alert if it exceeds a set threshold (e.g., 35 degrees).The alert information is then sent to a server, which then sends a notification to multiple smartphones saying, "The temperature inside the vehicle is dangerous. Please check immediately!"

[1369] Dialogue system and emotion engine using natural language processing technology

[1370] The server that receives the alert information activates a chatbot system using natural language processing technology. This system asks questions to the children and young children to check their situation. This invention also incorporates an emotion engine that analyzes the children's emotional state and adjusts the dialogue accordingly.

[1371] Specific examples

[1372] When an alert is issued, ChatGPT speaks to the child, saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" When the child responds, the emotion engine analyzes the response, and if the child appears anxious, it sends a reassuring message such as "It's okay, we're calling for help." If the child appears excited, it gives calming instructions such as "Try taking a slow, deep breath."

[1373] The chatbot asks additional questions based on the child's responses and emotional state, gradually understanding the situation. For example, it asks questions such as "Where are you now?" and "Are you locked in the car?" to confirm the situation. It gives instructions on how to escape or call for help, and gives specific instructions such as "Try to find the door lock button." The content of this conversation and emotional data are sent to a server, and are then sent to multiple registered smartphones as additional information.

[1374] Through the above steps, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine.

[1375] The processing flow will be explained below.

[1376] Processing for cameras with motion sensors

[1377] Step 1:

[1378] The device (camera) activates the human presence sensor and acquires data from inside the vehicle. The sensor data is periodically collected and stored in the internal memory.

[1379] Step 2:

[1380] The terminal (camera) monitors the signal to detect when the vehicle key has been removed. If it detects that the key has been removed, it immediately checks the human sensor data at that time.

[1381] Step 3:

[1382] The device (camera) uses motion sensor data to determine whether a person is present inside the vehicle, and if a person is detected, it issues an audio alert both inside and outside the vehicle.

[1383] Step 4:

[1384] The terminal (camera) sends alert information (camera ID, person detection information) indicating that a person has been detected to the server.

[1385] Step 5:

[1386] The server receives the alert information and immediately sends a push notification to multiple registered smartphone IDs.

[1387] Processing a portable device that monitors the temperature inside a vehicle

[1388] Step 6:

[1389] The terminal (mobile phone) activates the temperature sensor at regular intervals to obtain the temperature inside the vehicle. The obtained temperature data is stored in the internal memory.

[1390] Step 7:

[1391] The device (mobile phone) analyzes the temperature data stored in its internal memory and checks whether it exceeds a set threshold (e.g., 35 degrees).

[1392] Step 8:

[1393] The terminal (mobile phone) will issue an audio alert inside and outside the vehicle if the temperature exceeds a threshold.

[1394] Step 9:

[1395] The terminal (mobile phone) sends abnormal temperature alert information (mobile phone ID, temperature information) to the server.

[1396] Step 10:

[1397] The server receives the abnormal temperature alert information and sends a push notification to multiple registered smartphone IDs.

[1398] Dialogue processing using natural language processing technology and emotion engines

[1399] Step 11:

[1400] When the server receives the alert information, it activates a chatbot system that uses natural language processing technology.

[1401] Step 12:

[1402] The chatbot begins a conversation with the child based on pre-defined prompts, such as "Hello, camera number XX has detected an alert. XX-chan, can you hear me?"

[1403] Step 13:

[1404] When the user (child) responds, the system asks additional questions based on the content of the response, gradually understanding the situation. For example, it asks questions such as "Where are you now?" or "Are you locked in the car?"

[1405] Step 14:

[1406] The chatbot uses the child's responses and analysis data from the emotion engine to determine the child's emotional state. If the child is anxious, it sends a reassuring message, such as "It's okay, we're calling for help."

[1407] Step 15:

[1408] If a child becomes agitated, the chatbot will give them calming instructions such as, "Take slow, deep breaths and stay calm."

[1409] Step 16:

[1410] The chatbot gathers information about the situation and gives instructions on how to escape or call for help, such as sending specific instructions to the child, such as "Try to find the door lock button."

[1411] Step 17:

[1412] The chatbot transmits the content of the conversation and emotional data to the server.

[1413] Step 18:

[1414] The server then sends additional push notifications to the registered smartphone ID based on the collected situational information and emotion data, such as a notification that "A child is locked in a car. Please check immediately."

[1415] Through the above steps, the system of the present invention can ensure the safety of kindergarteners and young children through the collaboration of a camera with a motion sensor, a portable temperature monitoring device, natural language processing technology, and an emotion engine.

[1416] Example 2

[1417] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1418] The problem to be solved by this invention is to prevent accidents involving kindergarten children and small children being left behind in vehicles and ensure their safety. In particular, the object is to provide a system that quickly and accurately detects abnormalities and provides necessary notifications and dialogue when the vehicle engine is stopped and the key is removed, or when the temperature inside the vehicle reaches a dangerous level.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1420] In this invention, the server includes: means for receiving data from a motion sensor camera to detect the presence of a person inside the vehicle; means for issuing an audio alert if a person remains inside the vehicle when the vehicle engine is stopped and the key is removed; means for transmitting the alert information to a remote computer; means for transmitting notifications to multiple communication devices based on the transmitted alert information; means for receiving temperature data from a portable device that monitors the temperature inside the vehicle and detecting an abnormal temperature; means for issuing an audio alert when an abnormal temperature is detected; means for transmitting the alert information to the remote computer; means for conducting a dialogue using natural language processing technology; means for gathering information about the situation through the dialogue and providing instructions on how to call for help; and means for transmitting the information about the situation to the remote computer. This allows the server to quickly detect the presence of a child or toddler inside the vehicle and an abnormal temperature, and to respond immediately by transmitting a remote notification along with an audio alert. Furthermore, the server can use natural language processing technology to dialogue with the child to accurately understand the situation and reassure them.

[1421] A "camera with a human sensor" is a photographic device installed inside a vehicle and equipped with a sensor to detect human movement and presence.

[1422] The "means for receiving data" refers to a device or system for acquiring information transmitted from a camera with a motion sensor or a mobile device.

[1423] "Means for issuing an audio alert" refers to a function or device for issuing an audio warning when an abnormality is detected inside the vehicle.

[1424] The "means for transmitting alert information to an external communication device" refers to a function or system for transferring detected abnormality information to an external server or communication device using the Internet or other communication means.

[1425] The "means for sending notifications" is a system for sending warnings and notifications to multiple registered terminals based on alert information sent to an external communication device.

[1426] A "portable temperature monitoring device" is a portable device used to measure and monitor the temperature inside a vehicle.

[1427] "Means for detecting abnormal temperatures" refers to functions or systems that analyze data obtained from temperature monitoring equipment and detect abnormal temperatures that exceed a set threshold.

[1428] "Means for conducting dialogue using natural language processing technology" refers to a system that uses artificial intelligence technology to understand human language and conduct dialogue.

[1429] "Means for gathering information about the situation and providing instructions on how to seek help" is a system that understands the user's situation through dialogue and provides instructions on appropriate measures and actions.

[1430] The "means for transmitting status information to an external communication device" refers to a function or system for transferring information collected through dialogue to an external communication device or server.

[1431] As an embodiment of the present invention, a system is provided that combines various sensors and communication means to prevent accidents involving the abandonment of kindergarten children or small children in vehicles. The following describes in detail each component of this system and its operation.

[1432] Motion sensor camera operation

[1433] The device (camera with motion sensor) is installed inside the vehicle and is equipped with a sensor that detects the presence of people. Specifically, a Raspberry Pi is used, to which a motion sensor is attached. The camera periodically collects sensor data and detects the movement of people inside the vehicle. When the vehicle engine is turned off and the key is removed, the device checks the motion sensor data and issues an audio alert if anyone remains inside the vehicle. The alert information is sent to a remote computer such as AWS Lambda, which then uses Firebase Cloud Messaging to send notifications to multiple registered communication devices.

[1434] Specific examples

[1435] For example, if the vehicle engine is turned off and the driver removes the key, the device (camera) will detect this action. If a child remains in the vehicle, the human sensor will detect their presence and issue a voice alert saying, "Caution! There is someone left in the vehicle!" The alert information is then sent to the server, and a push notification saying, "Emergency! There is a child left in the vehicle!" is sent to multiple registered smartphones.

[1436] Operation of a portable device for monitoring temperature inside a vehicle

[1437] The terminal (portable device) is used to monitor the temperature inside the vehicle. Specifically, it uses an Arduino with a temperature sensor attached. The portable device periodically measures the temperature inside the vehicle and issues an audio alert if it exceeds a threshold preset by the user. The alert information is sent to a server, which then sends notifications to multiple communication devices.

[1438] Specific examples

[1439] For example, if the temperature inside a vehicle reaches 40 degrees on a hot summer day, the terminal (mobile device) detects this and issues a voice alert saying, "The temperature inside the vehicle is very high!" The alert information is then sent to the server, which then sends a notification to multiple registered smartphones saying, "Temperature warning! The temperature inside the vehicle is dangerous, please check immediately!"

[1440] Dialogue system and emotion engine operation using natural language processing technology

[1441] The server that receives the alert information uses ChatGPT to launch a dialogue system using natural language processing technology. This system asks questions to the children and toddlers to confirm their situation. This invention also incorporates an emotion engine such as Affectiva to analyze the children's emotional state and adjust the dialogue accordingly.

[1442] Specific examples

[1443] For example, when an alert is issued, ChatGPT will speak to the child, saying, "Hello, camera number XX has detected an alert. Can you hear me, XX-chan?" If the child responds with "Help me!", the emotion engine will analyze the response and determine that the child is in an anxious state. The dialogue system will then send a message saying, "It's okay, I'm calling for help. Can you tell me where you are?" This information is stored on the server and sent to the parent's smartphone as additional information.

[1444] Prompt Sentence Examples

[1445] "If a child remains in the car, how do you speak to them, check in on them, and provide them with the support they need?"

[1446] In this way, the system of the present invention can ensure the safety of kindergarten children and young children by linking a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, and an emotion engine. Each component is realized using specific hardware and software, and the results are notified to parents and guardians in real time, allowing for quick and appropriate response.

[1447] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1448] Processing for cameras with motion sensors

[1449] Step 1:

[1450] The device (camera) periodically collects data from the human presence sensor. In this case, the input is data on the presence and movement of people detected by the sensor. Specifically, a Raspberry Pi is used to acquire data from the sensor at regular intervals. The output is information on whether a person has been detected.

[1451] Step 2:

[1452] The terminal (camera) detects when the vehicle engine is stopped and the key is removed. The input is engine status information obtained from the vehicle's internal system. The output is a signal indicating that the engine has been stopped and the key has been removed. For example, the terminal can monitor the engine status in cooperation with the vehicle's system and recognize that the key has been removed.

[1453] Step 3:

[1454] The terminal (camera) checks the motion sensor data when the engine is stopped and determines whether or not there is anyone left in the vehicle. The input is the motion sensor data acquired in step 1 and the engine stop information in step 2. The output is a determination result of whether or not there is anyone left in the vehicle. Specifically, the sensor data is analyzed, and if there is anyone left, the system proceeds to the next operation.

[1455] Step 4:

[1456] If the device (camera) determines that a person remains in the vehicle, it issues a voice alert through the speaker. The input is the judgment result obtained in step 3. The output is an audio alert. For example, it may issue a message saying, "Caution! There is a person remaining in the vehicle!"

[1457] Step 5:

[1458] The device (camera) issues an audio alert and simultaneously sends the alert information to the server. The input is the audio alert information from step 4. The output is the alert information that is sent to a remote computer. Specifically, the alert information is sent to the cloud using AWS Lambda.

[1459] Step 6:

[1460] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message saying "Emergency! A child is left in the car!" is sent to the smartphone via Firebase Cloud Messaging.

[1461] Processing a portable device that monitors the temperature inside a vehicle

[1462] Step 1:

[1463] The terminal (mobile device) periodically measures the temperature inside the car. The input is data from the temperature sensor. Specifically, the temperature is periodically acquired using Arduino. The output is the current temperature information inside the car.

[1464] Step 2:

[1465] The terminal (mobile device) checks the temperature threshold set by the user in advance. The input is the threshold information set in the application. The output is the set threshold itself. Specifically, the threshold is set to 35 degrees in the application.

[1466] Step 3:

[1467] The terminal (portable device) compares the temperature data with a threshold value and detects abnormal temperatures. The inputs are the temperature information from step 1 and the threshold value information from step 2. The output is a determination result as to whether the temperature has exceeded the threshold value. Specifically, if the temperature inside the vehicle reaches 40 degrees, it is determined to be abnormal.

[1468] Step 4:

[1469] If the terminal (mobile device) detects an abnormal temperature, it issues a voice alert. The input is the judgment result of step 3. The output is an audio alert. For example, it may issue a warning saying, "The temperature inside the car is very high!"

[1470] Step 5:

[1471] The terminal (mobile device) issues a voice alert and simultaneously transmits the alert information to the server. The input is the alert information from step 4. The output is the alert information to be sent to a remote computer. Specifically, the terminal transmits information about abnormal temperature detection to the server.

[1472] Step 6:

[1473] The server sends a notification to multiple registered communication devices based on the received alert information. The input is the alert information sent to the server in step 5. The output is a push notification sent to each communication device. Specifically, a message is sent to the smartphone saying, "Temperature warning! The temperature inside the car is dangerous, please check immediately!"

[1474] Dialogue systems and emotion engines using natural language processing technology

[1475] Step 1:

[1476] The server launches a dialogue system using natural language processing technology based on the alert information received from the device. The input is the alert information. The output is that the dialogue system is ready to launch. Specifically, it launches a chatbot using ChatGPT.

[1477] Step 2:

[1478] The dialogue system launched by the server speaks to the kindergartener or toddler. The input is the initial question to the kindergartener. The output is a message to the kindergartener saying, "Hello, camera number XX has detected an alert. XX-chan, can you hear me?" Specifically, the system generates a message to gently inquire about the situation for the child.

[1479] Step 3:

[1480] The server receives the child's response and analyzes their emotional state using an emotion engine. The input is the child's response. The output is the analysis result of the child's emotional state. Specifically, the response "Help me!" is received and analyzed as an anxious state.

[1481] Step 4:

[1482] The server adjusts the dialogue content according to the emotional state. The input is the analysis result of the emotion engine. The output is an appropriately adjusted dialogue message. Specifically, it sends the message, "I'm OK, I'm calling for help. Can you tell me where you are?"

[1483] Step 5:

[1484] The server then provides more appropriate instructions based on the situational information collected through the dialogue. The input is the child's location and situation. The output is additional, more specific instructions. For example, the server might give instructions such as, "Try to find the door lock button."

[1485] Step 6:

[1486] The server notifies the registered communication devices of the content of the conversation and emotional data. The input is the content of the conversation and the emotional data. The output is a push notification with additional information. Specifically, a message is sent to the parent's smartphone saying, "Your child is asking for help. Please tell them to find the door lock button."

[1487] Through the above specific processing steps, the system of the present invention realizes a quick and accurate response to ensure the safety of kindergarten children and young children.

[1488] (Application example 2)

[1489] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1490] In recent years, as self-driving vehicles have become more common, accidents involving kindergarten children and young children being left behind inside the vehicle have become a problem. Such accidents are particularly likely to occur when the driver leaves the vehicle, posing a serious risk to the children's lives. Furthermore, the temperature inside the vehicle may rise, putting children at risk of heatstroke. Furthermore, if a child panics inside the vehicle, it becomes difficult to respond appropriately. For these reasons, a comprehensive system to ensure the safety of kindergarten children and young children is needed in self-driving vehicles.

[1491] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving data obtained from a motion sensor camera and detecting the presence of a person in the vehicle, means for issuing an audio alert if a person remains in the vehicle when the vehicle key is removed, means for transmitting the alert information to an external communication device, means for transmitting notifications to multiple registered terminals based on the transmitted alert information, means for conducting a dialogue using natural language processing technology, means for collecting information about the situation through the dialogue, means for analyzing responses using an emotion analysis engine and generating an appropriate message based on the responses, and means for instructing how to call for help based on the dialogue and analyzed information. This makes it possible to quickly and accurately grasp the situation of a kindergartener or young child left behind in a vehicle and safely deal with the situation.

[1492] A "camera with a human sensor" is a camera that detects the presence of people inside a vehicle and is a device that has a built-in human sensor.

[1493] A "vehicle key" is a physical or electronic key used to start or stop a vehicle engine.

[1494] "Voice alert" is a function that issues a voice warning if an abnormality is detected inside the vehicle.

[1495] The "external communication device" is a communication device installed outside the vehicle, which transmits and receives information via the Internet.

[1496] "Multiple terminals" refers to terminal devices connected to a communication network, such as smartphones and tablets owned by registered users.

[1497] "Natural language processing technology" is a technology for understanding human language and engaging in dialogue, and is a technology for analyzing and generating text.

[1498] An "emotion analysis engine" is a program that analyzes a user's emotional state based on data such as their statements and facial expressions.

[1499] "Dialogue means" refers to a means for communicating with a user, and is an interactive interface that uses voice or text.

[1500] "Instructions on how to call for help" refers to a function that provides specific instructions that children and young children need to call for help while in a vehicle.

[1501] MODE FOR CARRYING OUT THE INVENTION

[1502] The system of the present invention is designed to prevent accidents involving the abandonment of kindergarten children and young children in self-driving vehicles. The system consists of the following main components: a camera with a motion sensor, a portable temperature monitoring device, a dialogue system using natural language processing technology, an emotion analysis engine, and a server-based system.

[1503] System Configuration

[1504] 1. Motion Sensor Camera:

[1505] The terminal, a motion sensor camera, is installed inside the vehicle and periodically collects sensor data. This camera has the function of detecting whether there is a person inside the vehicle and confirming the presence of a person when the car key is removed. If there is a person inside, it issues an audio alert and sends the corresponding alert information to the server.

[1506] 2. Portable temperature monitoring device:

[1507] The terminal, a portable temperature monitoring device, monitors the temperature inside the vehicle and issues an audio alert if the temperature exceeds a set threshold. This information is also sent to a server, which then sends notifications to multiple terminals.

[1508] 3. Dialogue systems using natural language processing technology:

[1509] The server activates a chatbot using natural language processing technology to converse with the children in the car. The system asks questions to check the children's status and analyzes their responses using an emotion analysis engine.

[1510] 4. Sentiment Analysis Engine:

[1511] The emotion analysis engine analyzes the emotional state of the children during the conversation and guides them to an appropriate response accordingly, a process that is important for reducing the anxiety and panic that the children may be feeling.

[1512] 5. Server and Notification System:

[1513] The server aggregates alert information and the content of conversations between the children and the nursery school, and sends notifications to multiple devices via an external communication device, allowing relevant parties to quickly grasp the situation and take appropriate action.

[1514] The specific hardware and software used

[1515] Camera and motion sensor: Using development kits such as Arduino, a dedicated camera is installed inside the vehicle.

[1516] Temperature sensor: Connect a temperature sensor to the Raspberry Pi to obtain data in real time.

[1517] Server: Uses cloud services such as AWS and Google Cloud Platform.

[1518] Natural language processing technology: A dialogue system using OpenAI's GPT-3.5.

[1519] Sentiment analysis engine: Uses Microsoft Azure's emotion recognition API, etc.

[1520] Specific examples

[1521] For example, suppose a vehicle's engine shuts off on a hot summer day, and the driver leaves the vehicle and removes the key. If a child is left inside the vehicle, a motion sensor camera will detect the child's presence and issue an audio alert. At the same time, a portable temperature monitoring device will detect that the temperature inside the vehicle has exceeded 35 degrees and also issue an audio alert. This alert information is sent to a server, and push notifications are sent to multiple registered smartphones, stating, "There is a child left inside the vehicle!" and "The temperature inside the vehicle is dangerous, please check immediately!"

[1522] The server then launches a dialogue system using natural language processing technology and begins asking questions to the child, such as "Hello, Mr. / Ms. X, can you hear me?" If the child responds, the emotion analysis engine analyzes the response and, if the child seems anxious, offers reassuring words like "It's okay, I'm calling for help."

[1523] Prompt Sentence Examples

[1524] Child: "Yes, I can hear you."

[1525] Dialogue system: "Hello, Mr. / Ms. X. We've detected an in-car alert. Where are you now?"

[1526] Kindergartener: "I'm near the door."

[1527] Dialogue system: "Don't worry. Someone will come to help you shortly. Try finding the door lock button."

[1528] In this way, the system of the present invention can take comprehensive measures to ensure child safety.

[1529] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1530] Step 1:

[1531] The terminal collects data inside the vehicle using a camera with a motion sensor and a portable temperature monitoring device. The motion sensor detects whether a person is inside the vehicle, and the portable temperature monitoring device monitors the temperature inside the vehicle. The data obtained from these sensors (input) is sent to the terminal as human presence and temperature information (output).

[1532] Step 2:

[1533] The terminal processes the collected data and makes a decision based on the decision criteria. Specifically, if the motion sensor detects a person inside the vehicle, or if the temperature sensor exceeds a set threshold (e.g., 35 degrees), an audio alert is issued. The terminal then issues an audio alert and sends the information to the server (input data: presence of a person, temperature information; output data: alert information).

[1534] Step 3:

[1535] The server receives alert information sent from the device. The received alert information is about the presence of people and abnormal temperatures (input). Based on this information, the server sends push notifications to multiple registered devices (output). At this stage, the alert information is delivered to the user's smartphone or tablet.

[1536] Step 4:

[1537] Upon receiving the alert information, the server activates a dialogue system using natural language processing technology. This dialogue system uses a generative AI model to generate appropriate dialogue prompts for kindergarteners and young children (input data: alert information; output data: dialogue prompts).

[1538] Step 5:

[1539] The user (a child or toddler) responds to prompts from the dialogue system. The user's response (input) is received in real time by the dialogue system. The server analyzes this response using a generative AI model and generates an appropriate reply (output).

[1540] Step 6:

[1541] The server uses an emotion analysis engine to analyze the user's emotional state from their response. Based on the analysis results, the server adjusts the content of the dialogue (input data: user's response; output data: emotion analysis results). For example, if the user shows signs of anxiety, it generates a reassuring message.

[1542] Step 7:

[1543] The server takes into account the dialogue and the results of sentiment analysis to generate specific instructions for asking for help. It generates prompts for appropriate actions along with the dialogue content and provides them to the user (input data: dialogue history, sentiment analysis results; output data: specific instructions). For example, it may give specific instructions such as "Try to find the door lock button."

[1544] Step 8:

[1545] The server sends dialogue and emotion analysis data to an external communication device and notifies multiple registered devices of the situation (input data: dialogue history, emotion analysis results; output data: situation notification). This allows the relevant parties to grasp the situation in real time.

[1546] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1548] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1549] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1550] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1551] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1552] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1553] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balan...

Claims

1. Cameras with motion sensors, means for receiving data obtained from the camera and detecting the presence of a person in the vehicle; means for issuing an audio alert if a person remains in the vehicle when the vehicle key is removed; means for transmitting the alert information to an external communication device; means for transmitting a notification to a plurality of registered terminals based on the transmitted alert information; A system including:

2. a portable device for monitoring the temperature inside the vehicle; means for receiving temperature data obtained from the portable device and detecting abnormal temperatures; A means to issue an audio alert when an abnormal temperature is detected, means for transmitting the alert information to an external communication device; means for transmitting a notification to a plurality of registered terminals based on the transmitted alert information; The system of claim 1 , comprising:

3. a means for receiving alert information and engaging in dialogue using natural language processing technology; A means for collecting information about the situation through the dialogue and instructing how to seek help; means for transmitting the status information to an external communication device; The system of claim 1 , comprising:

Images