system

Abstract

Provide a system. [Solution] 1. A system for monitoring persons in a vehicle, comprising: A method for counting the number of people inside a vehicle using a camera equipped with facial recognition technology; means for acquiring temperature data from a thermometer installed in the vehicle; A means for acquiring location information of a name tag equipped with a GPS function; a means for detecting anomalies by integrating the facial recognition data, temperature data, and GPS data; means for issuing an alert based on the anomaly detection result; and means for notifying the mobile terminal of the issued alert.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there have been many accidents involving young children being left behind in vehicles during the extreme heat. This problem is not just a household issue, but also affects facilities that provide transportation, such as nurseries and day care centers. An effective system is needed to prevent such accidents. However, current systems have difficulty grasping the situation inside the vehicle in real time and responding quickly, which means that safety is not fully ensured. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides a system including the following means. First, a means is provided for counting the number of people inside the vehicle using a camera equipped with facial recognition technology. Next, a means is provided for acquiring temperature data from a thermometer installed inside the vehicle. Furthermore, a means is provided for acquiring location information from name tags equipped with GPS functionality. A means is provided for integrating these data to detect abnormalities, and a means is provided for issuing an alert based on the abnormality detection result. Finally, a system is provided that includes a means for notifying a mobile terminal of the issued alert. This makes it possible to detect abnormalities in the people and environment inside the vehicle in real time and respond promptly.

[0006] "Facial recognition technology" is a technology that detects human faces from digital images or video frames and recognizes them as specific people.

[0007] A "camera" is a device that records optical images and is used to take digital pictures and videos.

[0008] "Interior of vehicle" refers to the interior space of a motor vehicle or other means of transportation, typically including the seating and driver's seat.

[0009] A "thermometer" is a device that measures the temperature of an environment or object and displays it in digital or analog form.

[0010] "Temperature data" refers to the temperature measurement value obtained by a thermometer, and is information expressed in the form of a digital signal or the like.

[0011] "GPS function" means a function for determining the precise location of an object using the Global Positioning System.

[0012] A "name tag" is a portable device that carries identifying information about a person or object, usually with built-in GPS functionality.

[0013] "Location information" refers to geographic coordinate information of a specific location obtained using a GPS function.

[0014] "Data integration" is the process of bringing together information obtained from different sources in a unified, analyzable format.

[0015] "Anomaly detection" means automatically finding irregular situations or conditions that exceed set standards or conditions.

[0016] "Sending an alert" refers to the action of sending a warning or notification when an abnormality is detected.

[0017] "Mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0018] "Notifying" is the act of informing other devices or people of specific information. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system designed to ensure vehicle safety and prevent accidents involving young children being left behind in vehicles. The system utilizes face recognition technology, a thermometer, a GPS function, and a mobile device.

[0041] System configuration

[0042] 1. Camera with facial recognition technology

[0043] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[0044] 2. Vehicle interior temperature gauge

[0045] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[0046] 3. GPS-enabled name badges

[0047] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[0048] 4. Server

[0049] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[0050] 5. Mobile devices

[0051] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[0052] System operation example

[0053] Example 1: Detecting abnormal temperatures inside a vehicle

[0054] The car's interior thermometer periodically measures the temperature and sends the measurement results to the server.

[0055] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[0056] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[0057] The mobile device receives the alert and displays it to the user.

[0058] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car, immediately evacuating anyone in the car, etc.).

[0059] Example 2: Detecting objects left behind in a vehicle

[0060] After the engine is shut off, the server monitors the location of the GPS name tag for 10 minutes.

[0061] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[0062] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[0063] The mobile device receives the notification and displays it to the user.

[0064] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[0065] Example 3: Checking the number of people entering and leaving

[0066] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[0067] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[0068] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[0069] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[0070] The mobile device receives the notification and displays it to the user.

[0071] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0072] With this configuration and processing, the system can effectively monitor for dangers such as people being left behind in the vehicle or high temperatures, and respond quickly to prevent accidents before they occur.

[0073] The processing flow will be explained below.

[0074] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[0075] Step 1:

[0076] The interior thermometer periodically measures the temperature and sends the results to the server.

[0077] Step 2:

[0078] The server records the received temperature data in a database.

[0079] Step 3:

[0080] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0081] Step 4:

[0082] If the temperature remains above 40 degrees for 10 minutes, the server will determine that there is an abnormality.

[0083] Step 5:

[0084] If the server detects an abnormality, it will send an alert to the mobile device.

[0085] Step 6:

[0086] The mobile terminal receives the alert and displays a notification message to the user.

[0087] Step 7:

[0088] The user checks the notification on their mobile device and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0089] Example 2: Processing flow for detecting objects left behind in a vehicle

[0090] Step 1:

[0091] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0092] Step 2:

[0093] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0094] Step 3:

[0095] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0096] Step 4:

[0097] If the server detects an abnormality, it will send an alert to the mobile device.

[0098] Step 5:

[0099] The mobile terminal receives the alert and displays a notification message to the user.

[0100] Step 6:

[0101] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[0102] Example 3: Processing flow for checking the number of people entering and leaving

[0103] Step 1:

[0104] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0105] Step 2:

[0106] The server records the facial recognition data received at the time of boarding in a database.

[0107] Step 3:

[0108] When passengers disembark, the camera again uses facial recognition technology to count the number of passengers disembarking and sends the data to a server.

[0109] Step 4:

[0110] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[0111] Step 5:

[0112] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0113] Step 6:

[0114] If the server detects an abnormality, it will send an alert to the mobile device.

[0115] Step 7:

[0116] The mobile terminal receives the alert and displays a notification message to the user.

[0117] Step 8:

[0118] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[0119] As described above, a system is constructed that ensures safety inside the vehicle by performing specific operations at each processing step.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In recent years, the number of accidents inside vehicles has been increasing. In particular, serious accidents have occurred in which young children left behind inside vehicles suffer from heatstroke due to high temperatures. Such accidents occur because the situation inside the vehicle cannot be properly monitored even after the vehicle engine has been shut off. Another problem is the ineffectiveness of checking the number of passengers and early detection of abnormalities. Therefore, there is a need for effective monitoring of dangers such as people being left behind inside vehicles and high temperatures, and for rapid response.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from a name tag equipped with a GPS function, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for automatically starting the system when the vehicle engine is started. This makes it possible to effectively monitor for dangers such as people being left behind in the vehicle or high temperatures and to respond quickly.

[0125] "Facial recognition technology" is a technology that uses a camera to recognize a person's face and uses that data to identify individual people.

[0126] "Camera" means a device that captures images or videos and, if equipped with facial recognition technology, has the ability to detect and recognize a person's face.

[0127] An "in-vehicle thermometer" is a device that measures the temperature inside a vehicle and has the function of transmitting the measured data to a server.

[0128] A "name tag with GPS functionality" is a name tag equipped with GPS technology for acquiring location information and has the function of transmitting that location data to a server.

[0129] A "server" is a computer system that receives and analyzes various data, performs the necessary processing, and outputs the results.

[0130] The "abnormality detection means" has the function of analyzing facial recognition data, temperature data, and GPS data, and detecting an abnormality if the data exceeds a set standard.

[0131] The "alert issuing means" has a function of issuing a signal to notify an abnormality when it is detected.

[0132] A "mobile terminal" is a portable electronic device that has the function of receiving alerts from a server and notifying the user.

[0133] The "means for automatically starting the system when the vehicle engine is started" has the function of automatically activating the entire linked system when the vehicle engine is started.

[0134] The following describes an embodiment of the present invention. The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. This system works by linking face recognition technology, an in-vehicle thermometer, GPS functionality, and a mobile device.

[0135] System configuration

[0136] 1. Camera with facial recognition technology

[0137] Cameras will be installed inside the vehicles and will count people entering and exiting the vehicle using facial recognition technology. Facial recognition data will be used to identify people inside the vehicle and record the number of people entering and exiting the vehicle.

[0138] The server receives and analyzes the facial recognition data sent from the camera. For example, if a child and a parent are riding in the car, the data "child = 1, parent = 1" is sent to the server.

[0139] 2. Vehicle interior temperature gauge

[0140] The interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, allowing the vehicle's interior temperature to be monitored and a warning issued if a high temperature condition occurs.

[0141] The server analyzes the temperature data and switches to alert mode if the temperature exceeds a certain threshold (for example, 40 degrees). If the high temperature continues, the server recognizes it as an abnormality and sends out an alert.

[0142] 3. GPS-enabled name badges

[0143] The name tag periodically transmits the wearer's location to a server, which then compares the name tag's location data with the vehicle's GPS data to determine if anyone remains inside the vehicle.

[0144] The server detects the possibility of a name tag being left behind inside the vehicle, especially after the engine has been shut off, and sends out an alert.

[0145] 4. Server Roles

[0146] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert to the mobile device and records it in a database.

[0147] 5. Mobile devices

[0148] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user checks the notification and takes appropriate measures.

[0149] Example of operation

[0150] Example 1: Detecting abnormal temperatures inside a vehicle

[0151] The car's interior thermometer periodically measures the temperature and sends it to the server.

[0152] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[0153] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[0154] The mobile device receives the alert and displays it to the user.

[0155] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car or evacuating people in the car).

[0156] Example 2: Detecting objects left behind in a vehicle

[0157] After the engine is shut off, the server monitors the GPS name tag's location information for 10 minutes.

[0158] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[0159] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[0160] The mobile device receives the notification and displays it to the user.

[0161] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[0162] Example 3: Checking the number of people entering and leaving

[0163] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[0164] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[0165] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[0166] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[0167] The mobile device receives the notification and displays it to the user.

[0168] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0169] Prompt Sentence Examples

[0170] Example of an input prompt for a generative AI model:

[0171] Please provide a detailed explanation of each processing step of the in-vehicle safety system, including specific operations from starting the vehicle engine to notifying the user after an abnormality is detected.

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] Step 1: Boot the system

[0174] Input: Vehicle engine start signal

[0175] Specific operation: When the vehicle's engine starts, the server receives a start signal, which is sent from the vehicle's computer system to the server.

[0176] Output: The whole system starts up. The server checks the status of all sensors and cameras and logs their startup status.

[0177] Step 2: Facial recognition upon boarding

[0178] Input: Image data of the interior of the car captured by the camera

[0179] How it works: The camera captures real-time images of people entering the vehicle and uses a facial recognition algorithm to identify each person.

[0180] Data processing: The captured image data is analyzed using facial recognition technology and converted into person data.

[0181] Output: The person data is sent to the server, and the number of passengers and individual person information is recorded. For example, data such as "child = 1, parent = 1" is recorded.

[0182] Step 3: Regularly measure the interior temperature

[0183] Input: Temperature data sent from the car's interior thermometer

[0184] Specific operation: The car thermometer periodically measures the temperature inside the car and sends the data to the server, for example, every 5 minutes.

[0185] Data calculation: After receiving the temperature data, the server analyzes it and determines whether the temperature exceeds a certain threshold (e.g., 40 degrees).

[0186] Output: The analysis results are output, and if the temperature is abnormal, the system switches to alert mode. If the high temperature continues for 10 minutes, it is recognized as an abnormality and preparations begin to send an alert.

[0187] Step 4: Collect GPS name badge location data

[0188] Input: Location data sent from GPS name tag

[0189] Specific operation: The GPS name tag periodically transmits the location information of the person carrying it to a server.

[0190] Data processing: The server receives the nameplate location data and compares it with the GPS data inside the vehicle.

[0191] Output: After the engine is stopped, if the name tag is located inside the vehicle, the server detects the possibility that the person has been left behind and prepares to send an alert.

[0192] Step 5: Detect anomalies and send alerts

[0193] Input: Facial recognition data, temperature data, GPS data

[0194] Specific operation: The server integrates and analyzes the above data. If an abnormal condition is detected, an alert is sent immediately.

[0195] Data calculation: By integrating and analyzing each piece of data, an abnormality is detected if it exceeds set standards (for example, discrepancies in the number of passengers boarding and disembarking using facial recognition, high temperatures, or GPS data being left behind).

[0196] Output: If an anomaly is detected, the server sends an alert to the mobile device and also records it in the database.

[0197] Step 6: Notify users

[0198] Input: Alerts originating from the server

[0199] Specific behavior: The mobile device receives an alert from the server and displays a notification to the user, including the details of the anomaly and recommended actions.

[0200] Output: The user checks the notification and takes appropriate action. For example, if a high temperature warning is displayed, the user returns to the car and turns on the air conditioner, or evacuates people from the car.

[0201] These steps allow the system to ensure the safety of people inside the vehicle and respond quickly if something goes wrong.

[0202] (Application example 1)

[0203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0204] Currently, there has been an increase in accidents involving people being left behind in autonomous vehicles, particularly young children and the elderly. The danger of abnormally high interior temperatures has also been pointed out, and these problems need to be resolved quickly and effectively. However, conventional systems lack the means to quickly detect these abnormalities and issue appropriate alerts. In light of this situation, the present invention aims to ensure the safety of people in autonomous vehicles and prevent them from being left behind or experiencing abnormally high temperatures.

[0205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0206] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from name tags equipped with GPS functionality, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for managing boarding and alighting in cooperation with the control system of the autonomous vehicle. This makes it possible to monitor the safety of people inside the autonomous vehicle with high accuracy, and to immediately issue an alert and prompt appropriate action if an abnormality occurs.

[0207] "Facial recognition technology" is a technology that identifies and identifies a person's face from image data acquired by a camera.

[0208] A "camera" is a device installed inside a vehicle that uses facial recognition technology to capture image data of people getting on and off the vehicle.

[0209] "Counting people" refers to the operation of measuring the number of people present in the vehicle and sending that data to the server.

[0210] A "thermometer" is a device that measures the temperature inside a vehicle and sends the data to a server.

[0211] "Temperature data" is information about the temperature inside the vehicle measured by a thermometer.

[0212] The "GPS function" is a technology for obtaining geographical location information and is installed in the name tag.

[0213] A "name tag" is a portable device equipped with GPS functionality.

[0214] "Location Information" means geographic location data obtained using GPS functionality.

[0215] "Data integration" refers to the process of combining and analyzing facial recognition data, temperature data, and GPS data to determine the overall situation.

[0216] "Detecting anomalies" means analyzing the integrated data and discovering phenomena that deviate from normal conditions.

[0217] "Sending an alert" means sending a warning message based on a detected anomaly.

[0218] A "mobile terminal" is a portable device that receives alerts from the server and notifies the user.

[0219] An "autonomous vehicle" is a vehicle whose driving is automated by a system.

[0220] A "control system" is a computer system for managing and controlling the operation of a vehicle.

[0221] "Boarding and alighting management" is the process of managing the number of people boarding and alighting a vehicle using facial recognition technology.

[0222] The present invention is a system for ensuring safety inside a vehicle and preventing accidents, particularly those involving young children and the elderly, from occurring. The system operates by linking together facial recognition technology, a thermometer, a GPS function, a mobile device, and the control system of an autonomous vehicle.

[0223] System configuration

[0224] 1. Camera with facial recognition technology

[0225] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data obtained from the cameras is sent to a server.

[0226] 2. Vehicle interior temperature gauge

[0227] A thermometer installed inside the vehicle periodically measures the temperature inside the vehicle and transmits the data to a server.

[0228] 3. GPS-enabled name badges

[0229] The name tag is carried by the person and periodically transmits location information to a server, which receives this information and compares it with the vehicle's GPS data.

[0230] 4. Server

[0231] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[0232] 5. Mobile devices

[0233] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[0234] 6. Autonomous Vehicle Control Systems

[0235] The control system for autonomous vehicles receives instructions from the server to manage boarding and disembarking and respond to abnormal situations.

[0236] System operation example

[0237] Example 1: Detecting abnormal temperatures inside a vehicle

[0238] The thermometer periodically measures the temperature and sends the data to a server. The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees. If the high temperature continues for 10 minutes, the server determines that an abnormality has occurred and sends an alert to the mobile device. The user receives the alert and can take action, such as turning on the air conditioning in the car.

[0239] Example 2: Detecting objects left behind in a vehicle

[0240] After the engine is turned off, the server monitors the location information of the GPS name tag for 10 minutes. If the location information is inside the vehicle, it detects an abnormality and sends an alert to the mobile device. The user receives a notification and can quickly check whether anyone has been left inside the vehicle.

[0241] Example 3: Checking the number of people entering and leaving

[0242] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and sends the data to a server. When passengers exit the vehicle, the count is repeated and sent to the server. The server compares the number of passengers boarding and exiting the vehicle, and if there is a discrepancy, an anomaly is detected and an alert is sent to the mobile device.

[0243] Prompt Sentence Examples

[0244] Example: "Facial recognition technology was used in self-driving vehicles to prevent children from being left behind and to instantly notify mobile devices if temperatures were abnormally high. This system also utilized thermometer and GPS data for more effective monitoring. Consider implementing a similar safety monitoring system in your self-driving fleet."

[0245] Example prompt: "Give me an example of a design for a system to prevent people from being left behind in an autonomous vehicle. Explain specifically how the system combines facial recognition technology, an in-car thermometer, and GPS functionality."

[0246] This allows the user to grasp the current situation inside the vehicle in real time and take prompt action, significantly improving the safety of self-driving vehicles.

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] People counting using facial recognition

[0250] The server receives the video data sent from the camera. Then, using facial recognition technology, the server detects people's faces from this video data and counts the number of people. This data is recorded both when passengers board and when they exit the vehicle. The input is the video data, and the output is the counted number of people data. Specifically, the camera sends video of the interior of the vehicle in real time, which the server analyzes to accurately calculate the number of people inside the vehicle.

[0251] Step 2:

[0252] Acquiring temperature data

[0253] The server periodically receives temperature data from a thermometer installed inside the vehicle. It analyzes this temperature data and monitors the environmental conditions inside the vehicle. The input is the temperature data from the thermometer, and the output is the analyzed temperature information inside the vehicle. Specifically, the thermometer sends temperature data at regular intervals to the server, which then stores and analyzes the data.

[0254] Step 3:

[0255] Acquiring GPS data

[0256] The server periodically receives location information from the GPS-equipped name tag. This location information is matched with the vehicle's GPS data to confirm whether a person is present in the vehicle. The input is the GPS data from the name tag, and the output is the person's location information. Specifically, the name tag periodically transmits its location information to the server, which records and analyzes the data.

[0257] Step 4:

[0258] Data integration and analysis

[0259] The server integrates and analyzes facial recognition data, temperature data, and GPS data. It detects abnormalities based on this integrated data. The input is data from various sensors (facial recognition, temperature, GPS), and the output is the anomaly detection results. Specifically, the server comprehensively analyzes this data and detects abnormal conditions inside the vehicle.

[0260] Step 5:

[0261] Anomaly detection

[0262] The server analyzes the integrated data and determines whether an abnormality has occurred. For example, an abnormality is detected if the temperature inside the vehicle exceeds 40 degrees Celsius or if the location information of the name tag remains inside the vehicle after the engine is turned off. The input is the integrated data, and the output is the abnormality detection result and the issuance of an alert. Specifically, the server monitors the temperature data and GPS data, and detects an abnormality when certain conditions are met.

[0263] Step 6:

[0264] Sending alerts

[0265] If an anomaly is detected, the server generates an alert and sends it to the mobile device. The input is the anomaly detection result, and the output is the alert message. Specifically, when the server detects an anomaly, it immediately generates an alert message and sends it to the mobile device.

[0266] Step 7:

[0267] Mobile device notifications

[0268] The terminal receives an alert from the server and notifies the user in real time that an abnormality has occurred. The input is the alert message, and the output is a notification to the user. Specifically, when the terminal receives an alert, a pop-up notification or a voice notification is displayed to the user, urging them to take immediate action.

[0269] Step 8:

[0270] User response

[0271] The user checks the alert from the mobile device and takes the necessary action. For example, they turn on the air conditioner in the car or check the interior of the car. The input is the alert notification, and the output is the actual response action. Specifically, the user checks the alert and quickly operates the air conditioner or checks the interior of the car to ensure safety.

[0272] The above processing steps enable safety monitoring inside autonomous vehicles. By combining multiple technologies such as facial recognition, GPS, and temperature sensors, this system enhances safety inside the vehicle and is able to quickly detect and respond to abnormalities.

[0273] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0274] The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. The system works in conjunction with face recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[0275] System configuration

[0276] 1. Camera with facial recognition technology

[0277] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[0278] 2. Vehicle interior temperature gauge

[0279] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[0280] 3. GPS-enabled name badges

[0281] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[0282] 4. Emotion Engine

[0283] The emotion engine is installed on mobile devices and analyzes the user's facial expressions and voice to recognize their emotions, adjusting the alert content and urgency of notifications based on the recognized emotions.

[0284] 5. Server

[0285] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[0286] 6. Mobile Devices

[0287] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The emotion engine recognizes the user's emotions and adaptively adjusts the notification content. The user can check the notification content and take necessary measures.

[0288] System operation example

[0289] Example 1: Detecting abnormal temperatures inside a vehicle

[0290] The interior thermometer periodically measures the temperature and sends the results to the server.

[0291] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0292] If the server remains in a high temperature state for 10 minutes, it will be determined to be abnormal.

[0293] If the server detects an abnormality, it will send an alert to the mobile device.

[0294] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user is showing signs of stress or anxiety, the notification will be displayed with a higher level of urgency.

[0295] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0296] Example 2: Detecting objects left behind in a vehicle

[0297] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0298] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0299] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0300] If the server detects an abnormality, it will send an alert to the mobile device.

[0301] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user shows signs of impatience or nervousness, the notification will be displayed with a higher level of urgency.

[0302] The user checks the notification to see if anyone is left in the car.

[0303] Example 3: Checking the number of people entering and leaving

[0304] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0305] The server records the facial recognition data received at the time of boarding in a database.

[0306] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[0307] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[0308] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0309] If the server detects an abnormality, it will send an alert to the mobile device.

[0310] The mobile device receives the alert and uses an emotion engine to recognize the user's emotion. If the user is calm, a normal notification is displayed. If the user is upset, the notification is displayed with a higher level of urgency.

[0311] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0312] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[0313] The processing flow will be explained below.

[0314] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[0315] Step 1:

[0316] The interior thermometer periodically measures the temperature and sends the results to the server.

[0317] Step 2:

[0318] The server records the received temperature data in a database.

[0319] Step 3:

[0320] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0321] Step 4:

[0322] The server will determine that an abnormality has occurred if the temperature remains above 40 degrees for 10 minutes.

[0323] Step 5:

[0324] If the server detects an abnormality, it will send an alert to the mobile device.

[0325] Step 6:

[0326] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0327] Step 7:

[0328] The emotion engine analyzes the received data and determines whether the user is exhibiting stress or anxiety.

[0329] Step 8:

[0330] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0331] Step 9:

[0332] The user will check the notification and take appropriate action, such as improving the in-car environment or conducting rescue operations.

[0333] Example 2: Processing flow for detecting objects left behind in a vehicle

[0334] Step 1:

[0335] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0336] Step 2:

[0337] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0338] Step 3:

[0339] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0340] Step 4:

[0341] If the server detects an abnormality, it will send an alert to the mobile device.

[0342] Step 5:

[0343] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0344] Step 6:

[0345] The emotion engine analyzes the received data and determines whether the user is expressing impatience or tension.

[0346] Step 7:

[0347] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0348] Step 8:

[0349] The user will check the notification and act quickly to check if anyone is left in the vehicle.

[0350] Example 3: Processing flow for checking the number of people entering and leaving

[0351] Step 1:

[0352] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0353] Step 2:

[0354] The server records the facial recognition data received at the time of boarding in a database.

[0355] Step 3:

[0356] When passengers exit the vehicle, the onboard camera again uses facial recognition technology to count the number of people exiting the vehicle and sends the data to a server.

[0357] Step 4:

[0358] The server compares the number of passengers boarding and alighting to check whether there is a discrepancy in the numbers.

[0359] Step 5:

[0360] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0361] Step 6:

[0362] If the server detects an abnormality, it will send an alert to the mobile device.

[0363] Step 7:

[0364] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0365] Step 8:

[0366] The emotion engine analyzes the received data and determines whether the user is calm or upset.

[0367] Step 9:

[0368] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0369] Step 10:

[0370] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0371] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[0372] Example 2

[0373] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0374] The purpose of this invention is to ensure safety inside vehicles, particularly to prevent accidents involving young children being left behind inside. Conventional systems have difficulty properly monitoring and managing passenger entry and exit and the interior environment, posing particular risks to children left behind and high temperatures. To address these issues, a new method is needed that integrates facial recognition technology, a thermometer, GPS functionality, an emotion engine, and mobile devices.

[0375] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, a means for acquiring temperature data from a thermometer installed in the vehicle, a means for acquiring location information from a positioning device for acquiring location data, a means for detecting an abnormality by integrating the facial recognition data, the temperature data, and the location data, a means for transmitting an alarm signal based on the abnormality detection result, a means for notifying a mobile information terminal of the transmitted alarm signal, a means for acquiring emotion data using facial expression analysis technology, and a means for adjusting the content of the notification and the level of urgency based on the emotion data. This enables monitoring of the temperature inside the vehicle, detecting left-behind people, and providing appropriate notifications according to the user's emotional state.

[0376] "Facial recognition technology" is a technology that identifies a specific person from image data captured using a camera and obtains that information.

[0377] The "camera" is an image capturing device installed in a vehicle for detecting a person's face.

[0378] A "thermometer" is a measuring device that periodically measures the temperature inside a vehicle and acquires the data.

[0379] "Location data" is location information obtained from a positioning device with a GPS function, and is data that indicates the current location of a person or vehicle.

[0380] A "positioning device" is a device that has a GPS function, is carried by a person, and is used to obtain location data.

[0381] The "means for detecting abnormalities" refers to a method of analyzing facial recognition data, temperature data, and location data, and detecting abnormalities such as high temperatures or items left behind based on that data.

[0382] A "warning signal" is a signal that is sent from the server when an abnormality is detected, to alert the user.

[0383] A "personal digital assistant" is an electronic device that receives a warning signal sent from a server and notifies the user.

[0384] "Facial expression analysis technology" is a technology that analyzes facial expressions in order to recognize the emotional state of a user.

[0385] "Emotion data" is data relating to the user's emotional state obtained using facial expression analysis technology.

[0386] The present invention provides a system that ensures safety in vehicles and prevents accidents, particularly those involving young children being left behind in vehicles, by integrating facial recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[0387] Hardware and software used

[0388] 1. Camera with facial recognition technology

[0389] Example: Face recognition algorithm using OpenCV.

[0390] Processing: Cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle. The recognition data is sent to a server.

[0391] 2. Vehicle interior temperature gauge

[0392] Example: DHT22 sensor.

[0393] Processing: Periodically measure the temperature inside the car and send the data to a server.

[0394] 3. GPS-enabled name badges

[0395] Example: Garmin GPS units.

[0396] Process: The person carries the device and periodically sends location information to a server.

[0397] 4. Emotion Engine

[0398] Examples: Amazon Rekognition, Affectiva.

[0399] Processing: Installed on mobile devices, it analyzes the user's facial expressions and voice to recognize their emotional state.

[0400] 5. Server

[0401] Examples: AWS(registered trademark), Google(registered trademark) Cloud.

[0402] Processing: Detects abnormalities through integrated analysis of facial recognition data, temperature data, GPS data, and emotion data. If an abnormality is detected, an alarm signal is sent and a notification is sent to the mobile information terminal.

[0403] 6. Mobile Devices

[0404] Examples: iPhone (registered trademark), ANDROID (registered trademark) devices.

[0405] Processing: Receives a warning signal from the server, recognizes the user's emotions using the emotion engine, and adaptively adjusts the notification content.

[0406] System operation example

[0407] Example 1: Detecting abnormal temperatures inside a vehicle

[0408] The interior thermometer periodically measures the temperature and sends the results to the server.

[0409] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0410] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send out a warning signal.

[0411] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0412] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0413] Example 2: Detecting objects left behind in a vehicle

[0414] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0415] The server analyzes the name tag's location information to see if it is inside the vehicle, and if the location information remains inside the vehicle after 10 minutes, it determines that there is an abnormality.

[0416] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[0417] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0418] The user checks the notification to see if anyone is left in the car.

[0419] Example 3: Checking the number of people entering and leaving

[0420] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0421] The server records the facial recognition data received at the time of boarding in a database.

[0422] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[0423] The server compares the data at the time of boarding with the data at the time of disembarking, and if there is a difference in the number of people, it determines that there is an abnormality.

[0424] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[0425] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0426] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0427] Example prompts (for generative AI models)

[0428] I'm researching an in-vehicle safety system that combines facial recognition technology, thermometers, GPS functionality, an emotion engine, and mobile devices. I'd like to know more about the process and other related technologies for the following specific processing steps:

[0429] Step 1: Acquiring and sending facial recognition data

[0430] In this way, the system combines various sensors and technologies to monitor in-vehicle safety at a high level and respond quickly and appropriately in the event of an abnormality.

[0431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0432] Step 1: Acquire and send facial recognition data

[0433] The server receives facial recognition data from the in-car camera. The in-car camera is equipped with facial recognition technology using OpenCV to detect people getting in and out of the vehicle. This data is compiled as identification information by using a facial recognition algorithm to extract feature points from facial images captured by the camera. The server then receives this data and records it in a database. The input of the facial recognition data is the facial image captured by the camera, and the output is the facial identification information sent to the server.

[0434] Step 2: Acquire and analyze temperature data

[0435] The server receives temperature data from a thermometer installed inside the vehicle. A DHT22 sensor is used as this thermometer. The temperature sensor periodically measures the temperature inside the vehicle and generates temperature data. This data is analyzed in real time by the server. During analysis, the server monitors the measured temperature data and switches to high temperature warning mode if it exceeds a certain threshold (for example, 40 degrees). The input temperature data is the temperature information sent from the DHT22 sensor, and the output is the high temperature warning judgment result made by the server.

[0436] Step 3: Acquire and verify GPS data

[0437] The server periodically obtains location information from a name tag equipped with GPS functionality. It is assumed that a Garmin GPS unit is used. After the engine is turned off, the server continues to obtain GPS data for 10 minutes. This data is compared with the vehicle's GPS data to check whether the name tag is located inside the vehicle. If the GPS name tag is still inside the vehicle 10 minutes after the engine is turned off, it is determined to be an abnormality. The GPS data input is the location information obtained from the Garmin GPS unit, and the output is the abnormality detection result by the server.

[0438] Step 4: Obtaining and sending emotion data

[0439] The device uses an emotion engine to analyze the user's emotions. Technologies such as Amazon Rekognition and Affectiva are expected to be used. The user's facial expressions and voice are captured through the mobile device's camera and microphone, and an emotion analysis algorithm analyzes them to generate emotion data. This data is sent to a server and used to adjust the urgency and content of notifications. The emotion data input is the facial and voice information captured by the mobile device's camera and microphone, and the output is data sent to the server as the emotion analysis results.

[0440] Step 5: Detect anomalies and send warning signals

[0441] The server integrates and analyzes facial recognition data, temperature data, GPS data, and emotion data, thereby detecting abnormal conditions (high temperatures, passengers left behind, mismatch in passenger numbers) in real time. If an abnormality is detected, the server immediately sends out a warning signal and notifies the mobile information terminal. The inputs for data integration and analysis are facial recognition data, temperature data, GPS data, and emotion data, and the output is the abnormality detection result and a warning signal.

[0442] Step 6: Notify and respond to users

[0443] The device receives the transmitted warning signal and notifies the user. At this time, the emotion engine recognizes the user's emotions and adjusts the content and urgency of the notification. If the user shows signs of stress or anxiety, the notification is displayed with a higher level of urgency. The user checks the content of the notification and takes appropriate action (e.g., improving the environment inside the car or rescuing anyone stranded). The input for the notification and response is the warning signal sent from the server and the user's emotional data, and the output is the action taken by the user.

[0444] (Application example 2)

[0445] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] Conventional in-vehicle safety monitoring systems are limited to checking the number of passengers getting on and off and monitoring the temperature inside the vehicle, making it difficult to prevent accidents caused by human error or sudden temperature changes. Furthermore, they are unable to completely prevent accidents, particularly those involving young children or the elderly, from being left behind inside the vehicle. Furthermore, alert notifications do not take into account the emotional state of passengers, which can lead to delayed appropriate responses in emergencies.

[0447] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed in the vehicle, means for acquiring location information from a name-signing device equipped with a GPS function, means for recognizing people's emotions by analyzing their voices and facial expressions, means for detecting abnormalities by integrating the facial recognition data, temperature data, GPS data, and emotion data, means for issuing an alert based on the abnormality detection results, and means for notifying a computer terminal of the issued alert. This enables comprehensive monitoring of vehicle safety and prevents accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on passengers' emotional states enable immediate response even in emergencies.

[0448] "Facial recognition technology" is a technology that uses a camera to detect a person's face and identify that face as a specific individual.

[0449] A "thermometer" is a device that periodically measures the ambient temperature inside a vehicle and provides that data.

[0450] "GPS function" is a technology that uses a satellite positioning system to obtain location information on Earth.

[0451] A "name-registering device" is a device that is carried by a person and has the function of periodically transmitting location information.

[0452] "Means for recognizing a person's emotions by analyzing voice and facial expressions" refers to a technology that analyzes changes in a person's voice tone and facial expressions from audio data and video data to identify that person's emotional state.

[0453] "Means for detecting anomalies" refers to technology that integrates acquired facial recognition data, temperature data, GPS data, and emotional data, and uses the analysis results to detect unexpected events or dangerous situations.

[0454] "Means for sending alerts" refers to technology that sends information about an abnormality as audio, video, or a message when it is detected.

[0455] A "computer terminal" is an electronic device, such as a server or mobile device, that receives, processes, and displays data.

[0456] The present invention provides a system for ensuring safety in vehicles, particularly for monitoring the safety of young children and the elderly. The system operates by linking face recognition technology, a thermometer, a GPS function, an emotion analysis engine, and a computer terminal. Specific embodiments of the present invention are described below.

[0457] System configuration

[0458] 1. Camera with facial recognition technology

[0459] The cameras installed inside the vehicle recognize the faces of people getting on and off the vehicle and count the number of people. The facial recognition data is sent to a server for person detection and identification.

[0460] 2. Vehicle interior temperature gauge

[0461] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, which analyzes the temperature data and issues a warning if it detects an abnormally high temperature.

[0462] 3. Name-registering device with GPS function

[0463] The name registration device is carried by the person and periodically transmits its location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether the person remains in the vehicle after the engine has been turned off.

[0464] 4. Sentiment Analysis Engine

[0465] The emotion analysis engine is installed in the computer terminal and analyzes the user's voice and facial expressions to recognize emotions, and adjusts the alert content and urgency of notifications based on the recognized emotions.

[0466] 5. Server

[0467] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[0468] 6. Computer terminals

[0469] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. An emotion analysis engine is used to recognize the user's emotions and adaptively adjust the notification content. The user can then check the notification content and take necessary measures.

[0470] Hardware and software used

[0471] Camera: Equipped with a facial recognition camera

[0472] Thermometer: Vehicle interior thermometer

[0473] Named device: A device with GPS functionality

[0474] Server: Flask (framework), SQLAlchemy (object-relational mapping), SQLite (database)

[0475] Detection libraries: OpenCV (image processing), DeepFace (emotion analysis)

[0476] Computer terminals: smartphones, in-vehicle displays, etc.

[0477] Data processing and calculation

[0478] The server uses facial recognition technology to count the number of people inside the vehicle, encodes the data, and transmits it. Temperature and GPS data are collected in real time to monitor for abnormal conditions. An emotion analysis engine analyzes audio and video data to understand the user's emotional state. This data is integrated to detect abnormalities and send alerts at the appropriate time.

[0479] Prompt Sentence Examples

[0480] 1. Example prompt for facial recognition and emotion analysis:

[0481] "Please provide us with camera images. We will perform facial recognition and emotion analysis."

[0482] 2. Temperature data transmission prompt example:

[0483] Please enter the current interior temperature.

[0484] 3. Example of prompt for sending GPS data:

[0485] Please enter your current GPS location.

[0486] The above is a specific description of the embodiment of the invention. This system comprehensively monitors safety inside a vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, alert notifications are sent based on the emotional state of passengers, enabling a rapid response in emergencies. This significantly improves safety inside the vehicle and the user's sense of security.

[0487] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0488] Step 1:

[0489] A camera equipped with facial recognition technology captures the faces of people inside the vehicle. The input is the camera image, and the output is facial recognition data. This facial recognition data extracts facial feature points and identifies specific individuals. The server receives this facial recognition data and counts the number of people getting on and off the vehicle.

[0490] Step 2:

[0491] A thermometer installed inside the vehicle periodically collects temperature data. The input is the temperature data obtained from the sensor, and the output is a structured temperature value. The server receives this temperature data and monitors for abnormal temperatures.

[0492] Step 3:

[0493] The GPS-equipped device periodically transmits location information. The input is a GPS signal, and the output is latitude and longitude information. The server receives this location information and monitors the current location of the vehicle and the location of the passengers.

[0494] Step 4:

[0495] The server integrates facial recognition data, temperature data, and GPS data to detect anomalies. The input is this integrated data, and the output is the anomaly detection results. The server performs data analysis and detects abnormal conditions (e.g., high temperature or passengers left behind).

[0496] Step 5:

[0497] Voice and facial expressions are analyzed to recognize a person's emotions. A computer terminal analyzes the user's facial expressions and voice to generate emotion analysis data. The input is audio and video data, and the output is the identified emotional state. The server also integrates this emotion data to determine the urgency of the alert.

[0498] Step 6:

[0499] The server issues an alert based on the anomaly detection results. The input is the anomaly detection results and emotion data, and the output is an alert message. The server generates and sends an appropriate alert message according to the level of urgency.

[0500] Step 7:

[0501] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. The input is an alert message, and the output is a notification display. The terminal displays a notification to the user according to the urgency level, and the user takes appropriate measures based on this.

[0502] The above is a specific processing flow of the system of the present invention, clearly showing the input, data processing, and output at each step. This system comprehensively monitors the safety of the vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on the user's emotional state enable rapid response in emergencies.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] The present invention provides a system designed to ensure vehicle safety and prevent accidents involving young children being left behind in vehicles. The system utilizes face recognition technology, a thermometer, a GPS function, and a mobile device.

[0520] System configuration

[0521] 1. Camera with facial recognition technology

[0522] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[0523] 2. Vehicle interior temperature gauge

[0524] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[0525] 3. GPS-enabled name badges

[0526] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[0527] 4. Server

[0528] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[0529] 5. Mobile devices

[0530] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[0531] System operation example

[0532] Example 1: Detecting abnormal temperatures inside a vehicle

[0533] The car's interior thermometer periodically measures the temperature and sends the measurement results to the server.

[0534] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[0535] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[0536] The mobile device receives the alert and displays it to the user.

[0537] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car, immediately evacuating anyone in the car, etc.).

[0538] Example 2: Detecting objects left behind in a vehicle

[0539] After the engine is shut off, the server monitors the location of the GPS name tag for 10 minutes.

[0540] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[0541] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[0542] The mobile device receives the notification and displays it to the user.

[0543] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[0544] Example 3: Checking the number of people entering and leaving

[0545] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[0546] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[0547] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[0548] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[0549] The mobile device receives the notification and displays it to the user.

[0550] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0551] With this configuration and processing, the system can effectively monitor for dangers such as people being left behind in the vehicle or high temperatures, and respond quickly to prevent accidents before they occur.

[0552] The processing flow will be explained below.

[0553] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[0554] Step 1:

[0555] The interior thermometer periodically measures the temperature and sends the results to the server.

[0556] Step 2:

[0557] The server records the received temperature data in a database.

[0558] Step 3:

[0559] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0560] Step 4:

[0561] If the temperature remains above 40 degrees for 10 minutes, the server will determine that there is an abnormality.

[0562] Step 5:

[0563] If the server detects an abnormality, it will send an alert to the mobile device.

[0564] Step 6:

[0565] The mobile terminal receives the alert and displays a notification message to the user.

[0566] Step 7:

[0567] The user checks the notification on their mobile device and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0568] Example 2: Processing flow for detecting objects left behind in a vehicle

[0569] Step 1:

[0570] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0571] Step 2:

[0572] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0573] Step 3:

[0574] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0575] Step 4:

[0576] If the server detects an abnormality, it will send an alert to the mobile device.

[0577] Step 5:

[0578] The mobile terminal receives the alert and displays a notification message to the user.

[0579] Step 6:

[0580] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[0581] Example 3: Processing flow for checking the number of people entering and leaving

[0582] Step 1:

[0583] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0584] Step 2:

[0585] The server records the facial recognition data received at the time of boarding in a database.

[0586] Step 3:

[0587] When passengers disembark, the camera again uses facial recognition technology to count the number of passengers disembarking and sends the data to a server.

[0588] Step 4:

[0589] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[0590] Step 5:

[0591] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0592] Step 6:

[0593] If the server detects an abnormality, it will send an alert to the mobile device.

[0594] Step 7:

[0595] The mobile terminal receives the alert and displays a notification message to the user.

[0596] Step 8:

[0597] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[0598] As described above, a system is constructed that ensures safety inside the vehicle by performing specific operations at each processing step.

[0599] Example 1

[0600] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] In recent years, the number of accidents inside vehicles has been increasing. In particular, serious accidents have occurred in which young children left behind inside vehicles suffer from heatstroke due to high temperatures. Such accidents occur because the situation inside the vehicle cannot be properly monitored even after the vehicle engine has been shut off. Another problem is the ineffectiveness of checking the number of passengers and early detection of abnormalities. Therefore, there is a need for effective monitoring of dangers such as people being left behind inside vehicles and high temperatures, and for rapid response.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0603] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from a name tag equipped with a GPS function, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for automatically starting the system when the vehicle engine is started. This makes it possible to effectively monitor for dangers such as people being left behind in the vehicle or high temperatures and to respond quickly.

[0604] "Facial recognition technology" is a technology that uses a camera to recognize a person's face and uses that data to identify individual people.

[0605] "Camera" means a device that captures images or videos and, if equipped with facial recognition technology, has the ability to detect and recognize a person's face.

[0606] An "in-vehicle thermometer" is a device that measures the temperature inside a vehicle and has the function of transmitting the measured data to a server.

[0607] A "name tag with GPS functionality" is a name tag equipped with GPS technology for acquiring location information and has the function of transmitting that location data to a server.

[0608] A "server" is a computer system that receives and analyzes various data, performs the necessary processing, and outputs the results.

[0609] The "abnormality detection means" has the function of analyzing facial recognition data, temperature data, and GPS data, and detecting an abnormality if the data exceeds a set standard.

[0610] The "alert issuing means" has a function of issuing a signal to notify an abnormality when it is detected.

[0611] A "mobile terminal" is a portable electronic device that has the function of receiving alerts from a server and notifying the user.

[0612] The "means for automatically starting the system when the vehicle engine is started" has the function of automatically activating the entire linked system when the vehicle engine is started.

[0613] The following describes an embodiment of the present invention. The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. This system works by linking face recognition technology, an in-vehicle thermometer, GPS functionality, and a mobile device.

[0614] System configuration

[0615] 1. Camera with facial recognition technology

[0616] Cameras will be installed inside the vehicles and will count people entering and exiting the vehicle using facial recognition technology. Facial recognition data will be used to identify people inside the vehicle and record the number of people entering and exiting the vehicle.

[0617] The server receives and analyzes the facial recognition data sent from the camera. For example, if a child and a parent are riding in the car, the data "child = 1, parent = 1" is sent to the server.

[0618] 2. Vehicle interior temperature gauge

[0619] The interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, allowing the vehicle's interior temperature to be monitored and a warning issued if a high temperature condition occurs.

[0620] The server analyzes the temperature data and switches to alert mode if the temperature exceeds a certain threshold (for example, 40 degrees). If the high temperature continues, the server recognizes it as an abnormality and sends out an alert.

[0621] 3. GPS-enabled name badges

[0622] The name tag periodically transmits the wearer's location to a server, which then compares the name tag's location data with the vehicle's GPS data to determine if anyone remains inside the vehicle.

[0623] The server detects the possibility of a name tag being left behind inside the vehicle, especially after the engine has been shut off, and sends out an alert.

[0624] 4. Server Roles

[0625] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert to the mobile device and records it in a database.

[0626] 5. Mobile devices

[0627] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user checks the notification and takes appropriate measures.

[0628] Example of operation

[0629] Example 1: Detecting abnormal temperatures inside a vehicle

[0630] The car's interior thermometer periodically measures the temperature and sends it to the server.

[0631] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[0632] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[0633] The mobile device receives the alert and displays it to the user.

[0634] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car or evacuating people in the car).

[0635] Example 2: Detecting objects left behind in a vehicle

[0636] After the engine is shut off, the server monitors the GPS name tag's location information for 10 minutes.

[0637] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[0638] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[0639] The mobile device receives the notification and displays it to the user.

[0640] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[0641] Example 3: Checking the number of people entering and leaving

[0642] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[0643] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[0644] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[0645] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[0646] The mobile device receives the notification and displays it to the user.

[0647] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0648] Prompt Sentence Examples

[0649] Example of an input prompt for a generative AI model:

[0650] Please provide a detailed explanation of each processing step of the in-vehicle safety system, including specific operations from starting the vehicle engine to notifying the user after an abnormality is detected.

[0651] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0652] Step 1: Boot the system

[0653] Input: Vehicle engine start signal

[0654] Specific operation: When the vehicle's engine starts, the server receives a start signal, which is sent from the vehicle's computer system to the server.

[0655] Output: The whole system starts up. The server checks the status of all sensors and cameras and logs their startup status.

[0656] Step 2: Facial recognition upon boarding

[0657] Input: Image data of the interior of the car captured by the camera

[0658] How it works: The camera captures real-time images of people entering the vehicle and uses a facial recognition algorithm to identify each person.

[0659] Data processing: The captured image data is analyzed using facial recognition technology and converted into person data.

[0660] Output: The person data is sent to the server, and the number of passengers and individual person information is recorded. For example, data such as "child = 1, parent = 1" is recorded.

[0661] Step 3: Regularly measure the interior temperature

[0662] Input: Temperature data sent from the car's interior thermometer

[0663] Specific operation: The car thermometer periodically measures the temperature inside the car and sends the data to the server, for example, every 5 minutes.

[0664] Data calculation: After receiving the temperature data, the server analyzes it and determines whether the temperature exceeds a certain threshold (e.g., 40 degrees).

[0665] Output: The analysis results are output, and if the temperature is abnormal, the system switches to alert mode. If the high temperature continues for 10 minutes, it is recognized as an abnormality and preparations begin to send an alert.

[0666] Step 4: Collect GPS name badge location data

[0667] Input: Location data sent from GPS name tag

[0668] Specific operation: The GPS name tag periodically transmits the location information of the person carrying it to a server.

[0669] Data processing: The server receives the nameplate location data and compares it with the GPS data inside the vehicle.

[0670] Output: After the engine is stopped, if the name tag is located inside the vehicle, the server detects the possibility that the person has been left behind and prepares to send an alert.

[0671] Step 5: Detect anomalies and send alerts

[0672] Input: Facial recognition data, temperature data, GPS data

[0673] Specific operation: The server integrates and analyzes the above data. If an abnormal condition is detected, an alert is sent immediately.

[0674] Data calculation: By integrating and analyzing each piece of data, an abnormality is detected if it exceeds set standards (for example, discrepancies in the number of passengers boarding and disembarking using facial recognition, high temperatures, or GPS data being left behind).

[0675] Output: If an anomaly is detected, the server sends an alert to the mobile device and also records it in the database.

[0676] Step 6: Notify users

[0677] Input: Alerts originating from the server

[0678] Specific behavior: The mobile device receives an alert from the server and displays a notification to the user, including the details of the anomaly and recommended actions.

[0679] Output: The user checks the notification and takes appropriate action. For example, if a high temperature warning is displayed, the user returns to the car and turns on the air conditioner, or evacuates people from the car.

[0680] These steps allow the system to ensure the safety of people inside the vehicle and respond quickly if something goes wrong.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] Currently, there has been an increase in accidents involving people being left behind in autonomous vehicles, particularly young children and the elderly. The danger of abnormally high interior temperatures has also been pointed out, and these problems need to be resolved quickly and effectively. However, conventional systems lack the means to quickly detect these abnormalities and issue appropriate alerts. In light of this situation, the present invention aims to ensure the safety of people in autonomous vehicles and prevent them from being left behind or experiencing abnormally high temperatures.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from name tags equipped with GPS functionality, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for managing boarding and alighting in cooperation with the control system of the autonomous vehicle. This makes it possible to monitor the safety of people inside the autonomous vehicle with high accuracy, and to immediately issue an alert and prompt appropriate action if an abnormality occurs.

[0686] "Facial recognition technology" is a technology that identifies and identifies a person's face from image data acquired by a camera.

[0687] A "camera" is a device installed inside a vehicle that uses facial recognition technology to capture image data of people getting on and off the vehicle.

[0688] "Counting people" refers to the operation of measuring the number of people present in the vehicle and sending that data to the server.

[0689] A "thermometer" is a device that measures the temperature inside a vehicle and sends the data to a server.

[0690] "Temperature data" is information about the temperature inside the vehicle measured by a thermometer.

[0691] The "GPS function" is a technology for obtaining geographical location information and is installed in the name tag.

[0692] A "name tag" is a portable device equipped with GPS functionality.

[0693] "Location Information" means geographic location data obtained using GPS functionality.

[0694] "Data integration" refers to the process of combining and analyzing facial recognition data, temperature data, and GPS data to determine the overall situation.

[0695] "Detecting anomalies" means analyzing the integrated data and discovering phenomena that deviate from normal conditions.

[0696] "Sending an alert" means sending a warning message based on a detected anomaly.

[0697] A "mobile terminal" is a portable device that receives alerts from the server and notifies the user.

[0698] An "autonomous vehicle" is a vehicle whose driving is automated by a system.

[0699] A "control system" is a computer system for managing and controlling the operation of a vehicle.

[0700] "Boarding and alighting management" is the process of managing the number of people boarding and alighting a vehicle using facial recognition technology.

[0701] The present invention is a system for ensuring safety inside a vehicle and preventing accidents, particularly those involving young children and the elderly, from occurring. The system operates by linking together facial recognition technology, a thermometer, a GPS function, a mobile device, and the control system of an autonomous vehicle.

[0702] System configuration

[0703] 1. Camera with facial recognition technology

[0704] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data obtained from the cameras is sent to a server.

[0705] 2. Vehicle interior temperature gauge

[0706] A thermometer installed inside the vehicle periodically measures the temperature inside the vehicle and transmits the data to a server.

[0707] 3. GPS-enabled name badges

[0708] The name tag is carried by the person and periodically transmits location information to a server, which receives this information and compares it with the vehicle's GPS data.

[0709] 4. Server

[0710] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[0711] 5. Mobile devices

[0712] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[0713] 6. Autonomous Vehicle Control Systems

[0714] The control system for autonomous vehicles receives instructions from the server to manage boarding and disembarking and respond to abnormal situations.

[0715] System operation example

[0716] Example 1: Detecting abnormal temperatures inside a vehicle

[0717] The thermometer periodically measures the temperature and sends the data to a server. The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees. If the high temperature continues for 10 minutes, the server determines that an abnormality has occurred and sends an alert to the mobile device. The user receives the alert and can take action, such as turning on the air conditioning in the car.

[0718] Example 2: Detecting objects left behind in a vehicle

[0719] After the engine is turned off, the server monitors the location information of the GPS name tag for 10 minutes. If the location information is inside the vehicle, it detects an abnormality and sends an alert to the mobile device. The user receives a notification and can quickly check whether anyone has been left inside the vehicle.

[0720] Example 3: Checking the number of people entering and leaving

[0721] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and sends the data to a server. When passengers exit the vehicle, the count is repeated and sent to the server. The server compares the number of passengers boarding and exiting the vehicle, and if there is a discrepancy, an anomaly is detected and an alert is sent to the mobile device.

[0722] Prompt Sentence Examples

[0723] Example: "Facial recognition technology was used in self-driving vehicles to prevent children from being left behind and to instantly notify mobile devices if temperatures were abnormally high. This system also utilized thermometer and GPS data for more effective monitoring. Consider implementing a similar safety monitoring system in your self-driving fleet."

[0724] Example prompt: "Give me an example of a design for a system to prevent people from being left behind in an autonomous vehicle. Explain specifically how the system combines facial recognition technology, an in-car thermometer, and GPS functionality."

[0725] This allows the user to grasp the current situation inside the vehicle in real time and take prompt action, significantly improving the safety of self-driving vehicles.

[0726] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0727] Step 1:

[0728] People counting using facial recognition

[0729] The server receives the video data sent from the camera. Then, using facial recognition technology, the server detects people's faces from this video data and counts the number of people. This data is recorded both when passengers board and when they exit the vehicle. The input is the video data, and the output is the counted number of people data. Specifically, the camera sends video of the interior of the vehicle in real time, which the server analyzes to accurately calculate the number of people inside the vehicle.

[0730] Step 2:

[0731] Acquiring temperature data

[0732] The server periodically receives temperature data from a thermometer installed inside the vehicle. It analyzes this temperature data and monitors the environmental conditions inside the vehicle. The input is the temperature data from the thermometer, and the output is the analyzed temperature information inside the vehicle. Specifically, the thermometer sends temperature data at regular intervals to the server, which then stores and analyzes the data.

[0733] Step 3:

[0734] Acquiring GPS data

[0735] The server periodically receives location information from the GPS-equipped name tag. This location information is matched with the vehicle's GPS data to confirm whether a person is present in the vehicle. The input is the GPS data from the name tag, and the output is the person's location information. Specifically, the name tag periodically transmits its location information to the server, which records and analyzes the data.

[0736] Step 4:

[0737] Data integration and analysis

[0738] The server integrates and analyzes facial recognition data, temperature data, and GPS data. It detects abnormalities based on this integrated data. The input is data from various sensors (facial recognition, temperature, GPS), and the output is the anomaly detection results. Specifically, the server comprehensively analyzes this data and detects abnormal conditions inside the vehicle.

[0739] Step 5:

[0740] Anomaly detection

[0741] The server analyzes the integrated data and determines whether an abnormality has occurred. For example, an abnormality is detected if the temperature inside the vehicle exceeds 40 degrees Celsius or if the location information of the name tag remains inside the vehicle after the engine is turned off. The input is the integrated data, and the output is the abnormality detection result and the issuance of an alert. Specifically, the server monitors the temperature data and GPS data, and detects an abnormality when certain conditions are met.

[0742] Step 6:

[0743] Sending alerts

[0744] If an anomaly is detected, the server generates an alert and sends it to the mobile device. The input is the anomaly detection result, and the output is the alert message. Specifically, when the server detects an anomaly, it immediately generates an alert message and sends it to the mobile device.

[0745] Step 7:

[0746] Mobile device notifications

[0747] The terminal receives an alert from the server and notifies the user in real time that an abnormality has occurred. The input is the alert message, and the output is a notification to the user. Specifically, when the terminal receives an alert, a pop-up notification or a voice notification is displayed to the user, urging them to take immediate action.

[0748] Step 8:

[0749] User response

[0750] The user checks the alert from the mobile device and takes the necessary action. For example, they turn on the air conditioner in the car or check the interior of the car. The input is the alert notification, and the output is the actual response action. Specifically, the user checks the alert and quickly operates the air conditioner or checks the interior of the car to ensure safety.

[0751] The above processing steps enable safety monitoring inside autonomous vehicles. By combining multiple technologies such as facial recognition, GPS, and temperature sensors, this system enhances safety inside the vehicle and is able to quickly detect and respond to abnormalities.

[0752] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0753] The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. The system works in conjunction with face recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[0754] System configuration

[0755] 1. Camera with facial recognition technology

[0756] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[0757] 2. Vehicle interior temperature gauge

[0758] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[0759] 3. GPS-enabled name badges

[0760] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[0761] 4. Emotion Engine

[0762] The emotion engine is installed on mobile devices and analyzes the user's facial expressions and voice to recognize their emotions, adjusting the alert content and urgency of notifications based on the recognized emotions.

[0763] 5. Server

[0764] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[0765] 6. Mobile Devices

[0766] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The emotion engine recognizes the user's emotions and adaptively adjusts the notification content. The user can check the notification content and take necessary measures.

[0767] System operation example

[0768] Example 1: Detecting abnormal temperatures inside a vehicle

[0769] The interior thermometer periodically measures the temperature and sends the results to the server.

[0770] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0771] If the server remains in a high temperature state for 10 minutes, it will be determined to be abnormal.

[0772] If the server detects an abnormality, it will send an alert to the mobile device.

[0773] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user is showing signs of stress or anxiety, the notification will be displayed with a higher level of urgency.

[0774] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0775] Example 2: Detecting objects left behind in a vehicle

[0776] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0777] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0778] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0779] If the server detects an abnormality, it will send an alert to the mobile device.

[0780] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user shows signs of impatience or nervousness, the notification will be displayed with a higher level of urgency.

[0781] The user checks the notification to see if anyone is left in the car.

[0782] Example 3: Checking the number of people entering and leaving

[0783] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0784] The server records the facial recognition data received at the time of boarding in a database.

[0785] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[0786] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[0787] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0788] If the server detects an abnormality, it will send an alert to the mobile device.

[0789] The mobile device receives the alert and uses an emotion engine to recognize the user's emotion. If the user is calm, a normal notification is displayed. If the user is upset, the notification is displayed with a higher level of urgency.

[0790] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0791] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[0792] The processing flow will be explained below.

[0793] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[0794] Step 1:

[0795] The interior thermometer periodically measures the temperature and sends the results to the server.

[0796] Step 2:

[0797] The server records the received temperature data in a database.

[0798] Step 3:

[0799] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0800] Step 4:

[0801] The server will determine that an abnormality has occurred if the temperature remains above 40 degrees for 10 minutes.

[0802] Step 5:

[0803] If the server detects an abnormality, it will send an alert to the mobile device.

[0804] Step 6:

[0805] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0806] Step 7:

[0807] The emotion engine analyzes the received data and determines whether the user is exhibiting stress or anxiety.

[0808] Step 8:

[0809] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0810] Step 9:

[0811] The user will check the notification and take appropriate action, such as improving the in-car environment or conducting rescue operations.

[0812] Example 2: Processing flow for detecting objects left behind in a vehicle

[0813] Step 1:

[0814] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0815] Step 2:

[0816] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[0817] Step 3:

[0818] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[0819] Step 4:

[0820] If the server detects an abnormality, it will send an alert to the mobile device.

[0821] Step 5:

[0822] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0823] Step 6:

[0824] The emotion engine analyzes the received data and determines whether the user is expressing impatience or tension.

[0825] Step 7:

[0826] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0827] Step 8:

[0828] The user will check the notification and act quickly to check if anyone is left in the vehicle.

[0829] Example 3: Processing flow for checking the number of people entering and leaving

[0830] Step 1:

[0831] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0832] Step 2:

[0833] The server records the facial recognition data received at the time of boarding in a database.

[0834] Step 3:

[0835] When passengers exit the vehicle, the onboard camera again uses facial recognition technology to count the number of people exiting the vehicle and sends the data to a server.

[0836] Step 4:

[0837] The server compares the number of passengers boarding and alighting to check whether there is a discrepancy in the numbers.

[0838] Step 5:

[0839] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[0840] Step 6:

[0841] If the server detects an abnormality, it will send an alert to the mobile device.

[0842] Step 7:

[0843] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[0844] Step 8:

[0845] The emotion engine analyzes the received data and determines whether the user is calm or upset.

[0846] Step 9:

[0847] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[0848] Step 10:

[0849] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0850] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[0851] Example 2

[0852] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0853] The purpose of this invention is to ensure safety inside vehicles, particularly to prevent accidents involving young children being left behind inside. Conventional systems have difficulty properly monitoring and managing passenger entry and exit and the interior environment, posing particular risks to children left behind and high temperatures. To address these issues, a new method is needed that integrates facial recognition technology, a thermometer, GPS functionality, an emotion engine, and mobile devices.

[0854] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, a means for acquiring temperature data from a thermometer installed in the vehicle, a means for acquiring location information from a positioning device for acquiring location data, a means for detecting an abnormality by integrating the facial recognition data, the temperature data, and the location data, a means for transmitting an alarm signal based on the abnormality detection result, a means for notifying a mobile information terminal of the transmitted alarm signal, a means for acquiring emotion data using facial expression analysis technology, and a means for adjusting the content of the notification and the level of urgency based on the emotion data. This enables monitoring of the temperature inside the vehicle, detecting left-behind people, and providing appropriate notifications according to the user's emotional state.

[0855] "Facial recognition technology" is a technology that identifies a specific person from image data captured using a camera and obtains that information.

[0856] The "camera" is an image capturing device installed in a vehicle for detecting a person's face.

[0857] A "thermometer" is a measuring device that periodically measures the temperature inside a vehicle and acquires the data.

[0858] "Location data" is location information obtained from a positioning device with a GPS function, and is data that indicates the current location of a person or vehicle.

[0859] A "positioning device" is a device that has a GPS function, is carried by a person, and is used to obtain location data.

[0860] The "means for detecting abnormalities" refers to a method of analyzing facial recognition data, temperature data, and location data, and detecting abnormalities such as high temperatures or items left behind based on that data.

[0861] A "warning signal" is a signal that is sent from the server when an abnormality is detected, to alert the user.

[0862] A "personal digital assistant" is an electronic device that receives a warning signal sent from a server and notifies the user.

[0863] "Facial expression analysis technology" is a technology that analyzes facial expressions in order to recognize the emotional state of a user.

[0864] "Emotion data" is data relating to the user's emotional state obtained using facial expression analysis technology.

[0865] The present invention provides a system that ensures safety in vehicles and prevents accidents, particularly those involving young children being left behind in vehicles, by integrating facial recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[0866] Hardware and software used

[0867] 1. Camera with facial recognition technology

[0868] Example: Face recognition algorithm using OpenCV.

[0869] Processing: Cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle. The recognition data is sent to a server.

[0870] 2. Vehicle interior temperature gauge

[0871] Example: DHT22 sensor.

[0872] Processing: Periodically measure the temperature inside the car and send the data to a server.

[0873] 3. GPS-enabled name badges

[0874] Example: Garmin GPS units.

[0875] Process: The person carries the device and periodically sends location information to a server.

[0876] 4. Emotion Engine

[0877] Examples: Amazon Rekognition, Affectiva.

[0878] Processing: Installed on mobile devices, it analyzes the user's facial expressions and voice to recognize their emotional state.

[0879] 5. Server

[0880] Examples: AWS, Google Cloud.

[0881] Processing: Detects abnormalities through integrated analysis of facial recognition data, temperature data, GPS data, and emotion data. If an abnormality is detected, an alarm signal is sent and a notification is sent to the mobile information terminal.

[0882] 6. Mobile Devices

[0883] Example: iPhone, Android device.

[0884] Processing: Receives a warning signal from the server, recognizes the user's emotions using the emotion engine, and adaptively adjusts the notification content.

[0885] System operation example

[0886] Example 1: Detecting abnormal temperatures inside a vehicle

[0887] The interior thermometer periodically measures the temperature and sends the results to the server.

[0888] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[0889] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send out a warning signal.

[0890] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0891] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[0892] Example 2: Detecting objects left behind in a vehicle

[0893] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[0894] The server analyzes the name tag's location information to see if it is inside the vehicle, and if the location information remains inside the vehicle after 10 minutes, it determines that there is an abnormality.

[0895] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[0896] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0897] The user checks the notification to see if anyone is left in the car.

[0898] Example 3: Checking the number of people entering and leaving

[0899] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[0900] The server records the facial recognition data received at the time of boarding in a database.

[0901] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[0902] The server compares the data at the time of boarding with the data at the time of disembarking, and if there is a difference in the number of people, it determines that there is an abnormality.

[0903] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[0904] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[0905] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[0906] Example prompts (for generative AI models)

[0907] I'm researching an in-vehicle safety system that combines facial recognition technology, thermometers, GPS functionality, an emotion engine, and mobile devices. I'd like to know more about the process and other related technologies for the following specific processing steps:

[0908] Step 1: Acquiring and sending facial recognition data

[0909] In this way, the system combines various sensors and technologies to monitor in-vehicle safety at a high level and respond quickly and appropriately in the event of an abnormality.

[0910] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0911] Step 1: Acquire and send facial recognition data

[0912] The server receives facial recognition data from the in-car camera. The in-car camera is equipped with facial recognition technology using OpenCV to detect people getting in and out of the vehicle. This data is compiled as identification information by using a facial recognition algorithm to extract feature points from facial images captured by the camera. The server then receives this data and records it in a database. The input of the facial recognition data is the facial image captured by the camera, and the output is the facial identification information sent to the server.

[0913] Step 2: Acquire and analyze temperature data

[0914] The server receives temperature data from a thermometer installed inside the vehicle. A DHT22 sensor is used as this thermometer. The temperature sensor periodically measures the temperature inside the vehicle and generates temperature data. This data is analyzed in real time by the server. During analysis, the server monitors the measured temperature data and switches to high temperature warning mode if it exceeds a certain threshold (for example, 40 degrees). The input temperature data is the temperature information sent from the DHT22 sensor, and the output is the high temperature warning judgment result made by the server.

[0915] Step 3: Acquire and verify GPS data

[0916] The server periodically obtains location information from a name tag equipped with GPS functionality. It is assumed that a Garmin GPS unit is used. After the engine is turned off, the server continues to obtain GPS data for 10 minutes. This data is compared with the vehicle's GPS data to check whether the name tag is located inside the vehicle. If the GPS name tag is still inside the vehicle 10 minutes after the engine is turned off, it is determined to be an abnormality. The GPS data input is the location information obtained from the Garmin GPS unit, and the output is the abnormality detection result by the server.

[0917] Step 4: Obtaining and sending emotion data

[0918] The device uses an emotion engine to analyze the user's emotions. Technologies such as Amazon Rekognition and Affectiva are expected to be used. The user's facial expressions and voice are captured through the mobile device's camera and microphone, and an emotion analysis algorithm analyzes them to generate emotion data. This data is sent to a server and used to adjust the urgency and content of notifications. The emotion data input is the facial and voice information captured by the mobile device's camera and microphone, and the output is data sent to the server as the emotion analysis results.

[0919] Step 5: Detect anomalies and send warning signals

[0920] The server integrates and analyzes facial recognition data, temperature data, GPS data, and emotion data, thereby detecting abnormal conditions (high temperatures, passengers left behind, mismatch in passenger numbers) in real time. If an abnormality is detected, the server immediately sends out a warning signal and notifies the mobile information terminal. The inputs for data integration and analysis are facial recognition data, temperature data, GPS data, and emotion data, and the output is the abnormality detection result and a warning signal.

[0921] Step 6: Notify and respond to users

[0922] The device receives the transmitted warning signal and notifies the user. At this time, the emotion engine recognizes the user's emotions and adjusts the content and urgency of the notification. If the user shows signs of stress or anxiety, the notification is displayed with a higher level of urgency. The user checks the content of the notification and takes appropriate action (e.g., improving the environment inside the car or rescuing anyone stranded). The input for the notification and response is the warning signal sent from the server and the user's emotional data, and the output is the action taken by the user.

[0923] (Application example 2)

[0924] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0925] Conventional in-vehicle safety monitoring systems are limited to checking the number of passengers getting on and off and monitoring the temperature inside the vehicle, making it difficult to prevent accidents caused by human error or sudden temperature changes. Furthermore, they are unable to completely prevent accidents, particularly those involving young children or the elderly, from being left behind inside the vehicle. Furthermore, alert notifications do not take into account the emotional state of passengers, which can lead to delayed appropriate responses in emergencies.

[0926] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed in the vehicle, means for acquiring location information from a name-signing device equipped with a GPS function, means for recognizing people's emotions by analyzing their voices and facial expressions, means for detecting abnormalities by integrating the facial recognition data, temperature data, GPS data, and emotion data, means for issuing an alert based on the abnormality detection results, and means for notifying a computer terminal of the issued alert. This enables comprehensive monitoring of vehicle safety and prevents accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on passengers' emotional states enable immediate response even in emergencies.

[0927] "Facial recognition technology" is a technology that uses a camera to detect a person's face and identify that face as a specific individual.

[0928] A "thermometer" is a device that periodically measures the ambient temperature inside a vehicle and provides that data.

[0929] "GPS function" is a technology that uses a satellite positioning system to obtain location information on Earth.

[0930] A "name-registering device" is a device that is carried by a person and has the function of periodically transmitting location information.

[0931] "Means for recognizing a person's emotions by analyzing voice and facial expressions" refers to a technology that analyzes changes in a person's voice tone and facial expressions from audio data and video data to identify that person's emotional state.

[0932] "Means for detecting anomalies" refers to technology that integrates acquired facial recognition data, temperature data, GPS data, and emotional data, and uses the analysis results to detect unexpected events or dangerous situations.

[0933] "Means for sending alerts" refers to technology that sends information about an abnormality as audio, video, or a message when it is detected.

[0934] A "computer terminal" is an electronic device, such as a server or mobile device, that receives, processes, and displays data.

[0935] The present invention provides a system for ensuring safety in vehicles, particularly for monitoring the safety of young children and the elderly. The system operates by linking face recognition technology, a thermometer, a GPS function, an emotion analysis engine, and a computer terminal. Specific embodiments of the present invention are described below.

[0936] System configuration

[0937] 1. Camera with facial recognition technology

[0938] The cameras installed inside the vehicle recognize the faces of people getting on and off the vehicle and count the number of people. The facial recognition data is sent to a server for person detection and identification.

[0939] 2. Vehicle interior temperature gauge

[0940] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, which analyzes the temperature data and issues a warning if it detects an abnormally high temperature.

[0941] 3. Name-registering device with GPS function

[0942] The name registration device is carried by the person and periodically transmits its location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether the person remains in the vehicle after the engine has been turned off.

[0943] 4. Sentiment Analysis Engine

[0944] The emotion analysis engine is installed in the computer terminal and analyzes the user's voice and facial expressions to recognize emotions, and adjusts the alert content and urgency of notifications based on the recognized emotions.

[0945] 5. Server

[0946] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[0947] 6. Computer terminals

[0948] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. An emotion analysis engine is used to recognize the user's emotions and adaptively adjust the notification content. The user can then check the notification content and take necessary measures.

[0949] Hardware and software used

[0950] Camera: Equipped with a facial recognition camera

[0951] Thermometer: Vehicle interior thermometer

[0952] Named device: A device with GPS functionality

[0953] Server: Flask (framework), SQLAlchemy (object-relational mapping), SQLite (database)

[0954] Detection libraries: OpenCV (image processing), DeepFace (emotion analysis)

[0955] Computer terminals: smartphones, in-vehicle displays, etc.

[0956] Data processing and calculation

[0957] The server uses facial recognition technology to count the number of people inside the vehicle, encodes the data, and transmits it. Temperature and GPS data are collected in real time to monitor for abnormal conditions. An emotion analysis engine analyzes audio and video data to understand the user's emotional state. This data is integrated to detect abnormalities and send alerts at the appropriate time.

[0958] Prompt Sentence Examples

[0959] 1. Example prompt for facial recognition and emotion analysis:

[0960] "Please provide us with camera images. We will perform facial recognition and emotion analysis."

[0961] 2. Temperature data transmission prompt example:

[0962] Please enter the current interior temperature.

[0963] 3. Example of prompt for sending GPS data:

[0964] Please enter your current GPS location.

[0965] The above is a specific description of the embodiment of the invention. This system comprehensively monitors safety inside a vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, alert notifications are sent based on the emotional state of passengers, enabling a rapid response in emergencies. This significantly improves safety inside the vehicle and the user's sense of security.

[0966] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0967] Step 1:

[0968] A camera equipped with facial recognition technology captures the faces of people inside the vehicle. The input is the camera image, and the output is facial recognition data. This facial recognition data extracts facial feature points and identifies specific individuals. The server receives this facial recognition data and counts the number of people getting on and off the vehicle.

[0969] Step 2:

[0970] A thermometer installed inside the vehicle periodically collects temperature data. The input is the temperature data obtained from the sensor, and the output is a structured temperature value. The server receives this temperature data and monitors for abnormal temperatures.

[0971] Step 3:

[0972] The GPS-equipped device periodically transmits location information. The input is a GPS signal, and the output is latitude and longitude information. The server receives this location information and monitors the current location of the vehicle and the location of the passengers.

[0973] Step 4:

[0974] The server integrates facial recognition data, temperature data, and GPS data to detect anomalies. The input is this integrated data, and the output is the anomaly detection results. The server performs data analysis and detects abnormal conditions (e.g., high temperature or passengers left behind).

[0975] Step 5:

[0976] Voice and facial expressions are analyzed to recognize a person's emotions. A computer terminal analyzes the user's facial expressions and voice to generate emotion analysis data. The input is audio and video data, and the output is the identified emotional state. The server also integrates this emotion data to determine the urgency of the alert.

[0977] Step 6:

[0978] The server issues an alert based on the anomaly detection results. The input is the anomaly detection results and emotion data, and the output is an alert message. The server generates and sends an appropriate alert message according to the level of urgency.

[0979] Step 7:

[0980] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. The input is an alert message, and the output is a notification display. The terminal displays a notification to the user according to the urgency level, and the user takes appropriate measures based on this.

[0981] The above is a specific processing flow of the system of the present invention, clearly showing the input, data processing, and output at each step. This system comprehensively monitors the safety of the vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on the user's emotional state enable rapid response in emergencies.

[0982] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0983] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0984] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0985] [Third embodiment]

[0986] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0987] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0988] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0989] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0990] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0991] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0992] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0993] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0994] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0995] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0996] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0997] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0998] The present invention provides a system designed to ensure vehicle safety and prevent accidents involving young children being left behind in vehicles. The system utilizes face recognition technology, a thermometer, a GPS function, and a mobile device.

[0999] System configuration

[1000] 1. Camera with facial recognition technology

[1001] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[1002] 2. Vehicle interior temperature gauge

[1003] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[1004] 3. GPS-enabled name badges

[1005] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[1006] 4. Server

[1007] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[1008] 5. Mobile devices

[1009] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[1010] System operation example

[1011] Example 1: Detecting abnormal temperatures inside a vehicle

[1012] The car's interior thermometer periodically measures the temperature and sends the measurement results to the server.

[1013] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[1014] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[1015] The mobile device receives the alert and displays it to the user.

[1016] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car, immediately evacuating anyone in the car, etc.).

[1017] Example 2: Detecting objects left behind in a vehicle

[1018] After the engine is shut off, the server monitors the location of the GPS name tag for 10 minutes.

[1019] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[1020] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[1021] The mobile device receives the notification and displays it to the user.

[1022] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[1023] Example 3: Checking the number of people entering and leaving

[1024] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[1025] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[1026] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[1027] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[1028] The mobile device receives the notification and displays it to the user.

[1029] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1030] With this configuration and processing, the system can effectively monitor for dangers such as people being left behind in the vehicle or high temperatures, and respond quickly to prevent accidents before they occur.

[1031] The processing flow will be explained below.

[1032] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[1033] Step 1:

[1034] The interior thermometer periodically measures the temperature and sends the results to the server.

[1035] Step 2:

[1036] The server records the received temperature data in a database.

[1037] Step 3:

[1038] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1039] Step 4:

[1040] If the temperature remains above 40 degrees for 10 minutes, the server will determine that there is an abnormality.

[1041] Step 5:

[1042] If the server detects an abnormality, it will send an alert to the mobile device.

[1043] Step 6:

[1044] The mobile terminal receives the alert and displays a notification message to the user.

[1045] Step 7:

[1046] The user checks the notification on their mobile device and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1047] Example 2: Processing flow for detecting objects left behind in a vehicle

[1048] Step 1:

[1049] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1050] Step 2:

[1051] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1052] Step 3:

[1053] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1054] Step 4:

[1055] If the server detects an abnormality, it will send an alert to the mobile device.

[1056] Step 5:

[1057] The mobile terminal receives the alert and displays a notification message to the user.

[1058] Step 6:

[1059] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[1060] Example 3: Processing flow for checking the number of people entering and leaving

[1061] Step 1:

[1062] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1063] Step 2:

[1064] The server records the facial recognition data received at the time of boarding in a database.

[1065] Step 3:

[1066] When passengers disembark, the camera again uses facial recognition technology to count the number of passengers disembarking and sends the data to a server.

[1067] Step 4:

[1068] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[1069] Step 5:

[1070] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1071] Step 6:

[1072] If the server detects an abnormality, it will send an alert to the mobile device.

[1073] Step 7:

[1074] The mobile terminal receives the alert and displays a notification message to the user.

[1075] Step 8:

[1076] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[1077] As described above, a system is constructed that ensures safety inside the vehicle by performing specific operations at each processing step.

[1078] Example 1

[1079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1080] In recent years, the number of accidents inside vehicles has been increasing. In particular, serious accidents have occurred in which young children left behind inside vehicles suffer from heatstroke due to high temperatures. Such accidents occur because the situation inside the vehicle cannot be properly monitored even after the vehicle engine has been shut off. Another problem is the ineffectiveness of checking the number of passengers and early detection of abnormalities. Therefore, there is a need for effective monitoring of dangers such as people being left behind inside vehicles and high temperatures, and for rapid response.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1082] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from a name tag equipped with a GPS function, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for automatically starting the system when the vehicle engine is started. This makes it possible to effectively monitor for dangers such as people being left behind in the vehicle or high temperatures and to respond quickly.

[1083] "Facial recognition technology" is a technology that uses a camera to recognize a person's face and uses that data to identify individual people.

[1084] "Camera" means a device that captures images or videos and, if equipped with facial recognition technology, has the ability to detect and recognize a person's face.

[1085] An "in-vehicle thermometer" is a device that measures the temperature inside a vehicle and has the function of transmitting the measured data to a server.

[1086] A "name tag with GPS functionality" is a name tag equipped with GPS technology for acquiring location information and has the function of transmitting that location data to a server.

[1087] A "server" is a computer system that receives and analyzes various data, performs the necessary processing, and outputs the results.

[1088] The "abnormality detection means" has the function of analyzing facial recognition data, temperature data, and GPS data, and detecting an abnormality if the data exceeds a set standard.

[1089] The "alert issuing means" has a function of issuing a signal to notify an abnormality when it is detected.

[1090] A "mobile terminal" is a portable electronic device that has the function of receiving alerts from a server and notifying the user.

[1091] The "means for automatically starting the system when the vehicle engine is started" has the function of automatically activating the entire linked system when the vehicle engine is started.

[1092] The following describes an embodiment of the present invention. The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. This system works by linking face recognition technology, an in-vehicle thermometer, GPS functionality, and a mobile device.

[1093] System configuration

[1094] 1. Camera with facial recognition technology

[1095] Cameras will be installed inside the vehicles and will count people entering and exiting the vehicle using facial recognition technology. Facial recognition data will be used to identify people inside the vehicle and record the number of people entering and exiting the vehicle.

[1096] The server receives and analyzes the facial recognition data sent from the camera. For example, if a child and a parent are riding in the car, the data "child = 1, parent = 1" is sent to the server.

[1097] 2. Vehicle interior temperature gauge

[1098] The interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, allowing the vehicle's interior temperature to be monitored and a warning issued if a high temperature condition occurs.

[1099] The server analyzes the temperature data and switches to alert mode if the temperature exceeds a certain threshold (for example, 40 degrees). If the high temperature continues, the server recognizes it as an abnormality and sends out an alert.

[1100] 3. GPS-enabled name badges

[1101] The name tag periodically transmits the wearer's location to a server, which then compares the name tag's location data with the vehicle's GPS data to determine if anyone remains inside the vehicle.

[1102] The server detects the possibility of a name tag being left behind inside the vehicle, especially after the engine has been shut off, and sends out an alert.

[1103] 4. Server Roles

[1104] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert to the mobile device and records it in a database.

[1105] 5. Mobile devices

[1106] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user checks the notification and takes appropriate measures.

[1107] Example of operation

[1108] Example 1: Detecting abnormal temperatures inside a vehicle

[1109] The car's interior thermometer periodically measures the temperature and sends it to the server.

[1110] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[1111] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[1112] The mobile device receives the alert and displays it to the user.

[1113] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car or evacuating people in the car).

[1114] Example 2: Detecting objects left behind in a vehicle

[1115] After the engine is shut off, the server monitors the GPS name tag's location information for 10 minutes.

[1116] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[1117] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[1118] The mobile device receives the notification and displays it to the user.

[1119] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[1120] Example 3: Checking the number of people entering and leaving

[1121] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[1122] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[1123] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[1124] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[1125] The mobile device receives the notification and displays it to the user.

[1126] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1127] Prompt Sentence Examples

[1128] Example of an input prompt for a generative AI model:

[1129] Please provide a detailed explanation of each processing step of the in-vehicle safety system, including specific operations from starting the vehicle engine to notifying the user after an abnormality is detected.

[1130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1131] Step 1: Boot the system

[1132] Input: Vehicle engine start signal

[1133] Specific operation: When the vehicle's engine starts, the server receives a start signal, which is sent from the vehicle's computer system to the server.

[1134] Output: The whole system starts up. The server checks the status of all sensors and cameras and logs their startup status.

[1135] Step 2: Facial recognition upon boarding

[1136] Input: Image data of the interior of the car captured by the camera

[1137] How it works: The camera captures real-time images of people entering the vehicle and uses a facial recognition algorithm to identify each person.

[1138] Data processing: The captured image data is analyzed using facial recognition technology and converted into person data.

[1139] Output: The person data is sent to the server, and the number of passengers and individual person information is recorded. For example, data such as "child = 1, parent = 1" is recorded.

[1140] Step 3: Regularly measure the interior temperature

[1141] Input: Temperature data sent from the car's interior thermometer

[1142] Specific operation: The car thermometer periodically measures the temperature inside the car and sends the data to the server, for example, every 5 minutes.

[1143] Data calculation: After receiving the temperature data, the server analyzes it and determines whether the temperature exceeds a certain threshold (e.g., 40 degrees).

[1144] Output: The analysis results are output, and if the temperature is abnormal, the system switches to alert mode. If the high temperature continues for 10 minutes, it is recognized as an abnormality and preparations begin to send an alert.

[1145] Step 4: Collect GPS name badge location data

[1146] Input: Location data sent from GPS name tag

[1147] Specific operation: The GPS name tag periodically transmits the location information of the person carrying it to a server.

[1148] Data processing: The server receives the nameplate location data and compares it with the GPS data inside the vehicle.

[1149] Output: After the engine is stopped, if the name tag is located inside the vehicle, the server detects the possibility that the person has been left behind and prepares to send an alert.

[1150] Step 5: Detect anomalies and send alerts

[1151] Input: Facial recognition data, temperature data, GPS data

[1152] Specific operation: The server integrates and analyzes the above data. If an abnormal condition is detected, an alert is sent immediately.

[1153] Data calculation: By integrating and analyzing each piece of data, an abnormality is detected if it exceeds set standards (for example, discrepancies in the number of passengers boarding and disembarking using facial recognition, high temperatures, or GPS data being left behind).

[1154] Output: If an anomaly is detected, the server sends an alert to the mobile device and also records it in the database.

[1155] Step 6: Notify users

[1156] Input: Alerts originating from the server

[1157] Specific behavior: The mobile device receives an alert from the server and displays a notification to the user, including the details of the anomaly and recommended actions.

[1158] Output: The user checks the notification and takes appropriate action. For example, if a high temperature warning is displayed, the user returns to the car and turns on the air conditioner, or evacuates people from the car.

[1159] These steps allow the system to ensure the safety of people inside the vehicle and respond quickly if something goes wrong.

[1160] (Application example 1)

[1161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1162] Currently, there has been an increase in accidents involving people being left behind in autonomous vehicles, particularly young children and the elderly. The danger of abnormally high interior temperatures has also been pointed out, and these problems need to be resolved quickly and effectively. However, conventional systems lack the means to quickly detect these abnormalities and issue appropriate alerts. In light of this situation, the present invention aims to ensure the safety of people in autonomous vehicles and prevent them from being left behind or experiencing abnormally high temperatures.

[1163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1164] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from name tags equipped with GPS functionality, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for managing boarding and alighting in cooperation with the control system of the autonomous vehicle. This makes it possible to monitor the safety of people inside the autonomous vehicle with high accuracy, and to immediately issue an alert and prompt appropriate action if an abnormality occurs.

[1165] "Facial recognition technology" is a technology that identifies and identifies a person's face from image data acquired by a camera.

[1166] A "camera" is a device installed inside a vehicle that uses facial recognition technology to capture image data of people getting on and off the vehicle.

[1167] "Counting people" refers to the operation of measuring the number of people present in the vehicle and sending that data to the server.

[1168] A "thermometer" is a device that measures the temperature inside a vehicle and sends the data to a server.

[1169] "Temperature data" is information about the temperature inside the vehicle measured by a thermometer.

[1170] The "GPS function" is a technology for obtaining geographical location information and is installed in the name tag.

[1171] A "name tag" is a portable device equipped with GPS functionality.

[1172] "Location Information" means geographic location data obtained using GPS functionality.

[1173] "Data integration" refers to the process of combining and analyzing facial recognition data, temperature data, and GPS data to determine the overall situation.

[1174] "Detecting anomalies" means analyzing the integrated data and discovering phenomena that deviate from normal conditions.

[1175] "Sending an alert" means sending a warning message based on a detected anomaly.

[1176] A "mobile terminal" is a portable device that receives alerts from the server and notifies the user.

[1177] An "autonomous vehicle" is a vehicle whose driving is automated by a system.

[1178] A "control system" is a computer system for managing and controlling the operation of a vehicle.

[1179] "Boarding and alighting management" is the process of managing the number of people boarding and alighting a vehicle using facial recognition technology.

[1180] The present invention is a system for ensuring safety inside a vehicle and preventing accidents, particularly those involving young children and the elderly, from occurring. The system operates by linking together facial recognition technology, a thermometer, a GPS function, a mobile device, and the control system of an autonomous vehicle.

[1181] System configuration

[1182] 1. Camera with facial recognition technology

[1183] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data obtained from the cameras is sent to a server.

[1184] 2. Vehicle interior temperature gauge

[1185] A thermometer installed inside the vehicle periodically measures the temperature inside the vehicle and transmits the data to a server.

[1186] 3. GPS-enabled name badges

[1187] The name tag is carried by the person and periodically transmits location information to a server, which receives this information and compares it with the vehicle's GPS data.

[1188] 4. Server

[1189] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[1190] 5. Mobile devices

[1191] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[1192] 6. Autonomous Vehicle Control Systems

[1193] The control system for autonomous vehicles receives instructions from the server to manage boarding and disembarking and respond to abnormal situations.

[1194] System operation example

[1195] Example 1: Detecting abnormal temperatures inside a vehicle

[1196] The thermometer periodically measures the temperature and sends the data to a server. The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees. If the high temperature continues for 10 minutes, the server determines that an abnormality has occurred and sends an alert to the mobile device. The user receives the alert and can take action, such as turning on the air conditioning in the car.

[1197] Example 2: Detecting objects left behind in a vehicle

[1198] After the engine is turned off, the server monitors the location information of the GPS name tag for 10 minutes. If the location information is inside the vehicle, it detects an abnormality and sends an alert to the mobile device. The user receives a notification and can quickly check whether anyone has been left inside the vehicle.

[1199] Example 3: Checking the number of people entering and leaving

[1200] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and sends the data to a server. When passengers exit the vehicle, the count is repeated and sent to the server. The server compares the number of passengers boarding and exiting the vehicle, and if there is a discrepancy, an anomaly is detected and an alert is sent to the mobile device.

[1201] Prompt Sentence Examples

[1202] Example: "Facial recognition technology was used in self-driving vehicles to prevent children from being left behind and to instantly notify mobile devices if temperatures were abnormally high. This system also utilized thermometer and GPS data for more effective monitoring. Consider implementing a similar safety monitoring system in your self-driving fleet."

[1203] Example prompt: "Give me an example of a design for a system to prevent people from being left behind in an autonomous vehicle. Explain specifically how the system combines facial recognition technology, an in-car thermometer, and GPS functionality."

[1204] This allows the user to grasp the current situation inside the vehicle in real time and take prompt action, significantly improving the safety of self-driving vehicles.

[1205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1206] Step 1:

[1207] People counting using facial recognition

[1208] The server receives the video data sent from the camera. Then, using facial recognition technology, the server detects people's faces from this video data and counts the number of people. This data is recorded both when passengers board and when they exit the vehicle. The input is the video data, and the output is the counted number of people data. Specifically, the camera sends video of the interior of the vehicle in real time, which the server analyzes to accurately calculate the number of people inside the vehicle.

[1209] Step 2:

[1210] Acquiring temperature data

[1211] The server periodically receives temperature data from a thermometer installed inside the vehicle. It analyzes this temperature data and monitors the environmental conditions inside the vehicle. The input is the temperature data from the thermometer, and the output is the analyzed temperature information inside the vehicle. Specifically, the thermometer sends temperature data at regular intervals to the server, which then stores and analyzes the data.

[1212] Step 3:

[1213] Acquiring GPS data

[1214] The server periodically receives location information from the GPS-equipped name tag. This location information is matched with the vehicle's GPS data to confirm whether a person is present in the vehicle. The input is the GPS data from the name tag, and the output is the person's location information. Specifically, the name tag periodically transmits its location information to the server, which records and analyzes the data.

[1215] Step 4:

[1216] Data integration and analysis

[1217] The server integrates and analyzes facial recognition data, temperature data, and GPS data. It detects abnormalities based on this integrated data. The input is data from various sensors (facial recognition, temperature, GPS), and the output is the anomaly detection results. Specifically, the server comprehensively analyzes this data and detects abnormal conditions inside the vehicle.

[1218] Step 5:

[1219] Anomaly detection

[1220] The server analyzes the integrated data and determines whether an abnormality has occurred. For example, an abnormality is detected if the temperature inside the vehicle exceeds 40 degrees Celsius or if the location information of the name tag remains inside the vehicle after the engine is turned off. The input is the integrated data, and the output is the abnormality detection result and the issuance of an alert. Specifically, the server monitors the temperature data and GPS data, and detects an abnormality when certain conditions are met.

[1221] Step 6:

[1222] Sending alerts

[1223] If an anomaly is detected, the server generates an alert and sends it to the mobile device. The input is the anomaly detection result, and the output is the alert message. Specifically, when the server detects an anomaly, it immediately generates an alert message and sends it to the mobile device.

[1224] Step 7:

[1225] Mobile device notifications

[1226] The terminal receives an alert from the server and notifies the user in real time that an abnormality has occurred. The input is the alert message, and the output is a notification to the user. Specifically, when the terminal receives an alert, a pop-up notification or a voice notification is displayed to the user, urging them to take immediate action.

[1227] Step 8:

[1228] User response

[1229] The user checks the alert from the mobile device and takes the necessary action. For example, they turn on the air conditioner in the car or check the interior of the car. The input is the alert notification, and the output is the actual response action. Specifically, the user checks the alert and quickly operates the air conditioner or checks the interior of the car to ensure safety.

[1230] The above processing steps enable safety monitoring inside autonomous vehicles. By combining multiple technologies such as facial recognition, GPS, and temperature sensors, this system enhances safety inside the vehicle and is able to quickly detect and respond to abnormalities.

[1231] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1232] The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. The system works in conjunction with face recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[1233] System configuration

[1234] 1. Camera with facial recognition technology

[1235] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[1236] 2. Vehicle interior temperature gauge

[1237] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[1238] 3. GPS-enabled name badges

[1239] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[1240] 4. Emotion Engine

[1241] The emotion engine is installed on mobile devices and analyzes the user's facial expressions and voice to recognize their emotions, adjusting the alert content and urgency of notifications based on the recognized emotions.

[1242] 5. Server

[1243] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[1244] 6. Mobile Devices

[1245] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The emotion engine recognizes the user's emotions and adaptively adjusts the notification content. The user can check the notification content and take necessary measures.

[1246] System operation example

[1247] Example 1: Detecting abnormal temperatures inside a vehicle

[1248] The interior thermometer periodically measures the temperature and sends the results to the server.

[1249] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1250] If the server remains in a high temperature state for 10 minutes, it will be determined to be abnormal.

[1251] If the server detects an abnormality, it will send an alert to the mobile device.

[1252] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user is showing signs of stress or anxiety, the notification will be displayed with a higher level of urgency.

[1253] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1254] Example 2: Detecting objects left behind in a vehicle

[1255] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1256] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1257] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1258] If the server detects an abnormality, it will send an alert to the mobile device.

[1259] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user shows signs of impatience or nervousness, the notification will be displayed with a higher level of urgency.

[1260] The user checks the notification to see if anyone is left in the car.

[1261] Example 3: Checking the number of people entering and leaving

[1262] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1263] The server records the facial recognition data received at the time of boarding in a database.

[1264] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[1265] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[1266] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1267] If the server detects an abnormality, it will send an alert to the mobile device.

[1268] The mobile device receives the alert and uses an emotion engine to recognize the user's emotion. If the user is calm, a normal notification is displayed. If the user is upset, the notification is displayed with a higher level of urgency.

[1269] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1270] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[1271] The processing flow will be explained below.

[1272] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[1273] Step 1:

[1274] The interior thermometer periodically measures the temperature and sends the results to the server.

[1275] Step 2:

[1276] The server records the received temperature data in a database.

[1277] Step 3:

[1278] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1279] Step 4:

[1280] The server will determine that an abnormality has occurred if the temperature remains above 40 degrees for 10 minutes.

[1281] Step 5:

[1282] If the server detects an abnormality, it will send an alert to the mobile device.

[1283] Step 6:

[1284] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1285] Step 7:

[1286] The emotion engine analyzes the received data and determines whether the user is exhibiting stress or anxiety.

[1287] Step 8:

[1288] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1289] Step 9:

[1290] The user will check the notification and take appropriate action, such as improving the in-car environment or conducting rescue operations.

[1291] Example 2: Processing flow for detecting objects left behind in a vehicle

[1292] Step 1:

[1293] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1294] Step 2:

[1295] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1296] Step 3:

[1297] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1298] Step 4:

[1299] If the server detects an abnormality, it will send an alert to the mobile device.

[1300] Step 5:

[1301] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1302] Step 6:

[1303] The emotion engine analyzes the received data and determines whether the user is expressing impatience or tension.

[1304] Step 7:

[1305] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1306] Step 8:

[1307] The user will check the notification and act quickly to check if anyone is left in the vehicle.

[1308] Example 3: Processing flow for checking the number of people entering and leaving

[1309] Step 1:

[1310] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1311] Step 2:

[1312] The server records the facial recognition data received at the time of boarding in a database.

[1313] Step 3:

[1314] When passengers exit the vehicle, the onboard camera again uses facial recognition technology to count the number of people exiting the vehicle and sends the data to a server.

[1315] Step 4:

[1316] The server compares the number of passengers boarding and alighting to check whether there is a discrepancy in the numbers.

[1317] Step 5:

[1318] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1319] Step 6:

[1320] If the server detects an abnormality, it will send an alert to the mobile device.

[1321] Step 7:

[1322] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1323] Step 8:

[1324] The emotion engine analyzes the received data and determines whether the user is calm or upset.

[1325] Step 9:

[1326] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1327] Step 10:

[1328] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1329] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[1330] Example 2

[1331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1332] The purpose of this invention is to ensure safety inside vehicles, particularly to prevent accidents involving young children being left behind inside. Conventional systems have difficulty properly monitoring and managing passenger entry and exit and the interior environment, posing particular risks to children left behind and high temperatures. To address these issues, a new method is needed that integrates facial recognition technology, a thermometer, GPS functionality, an emotion engine, and mobile devices.

[1333] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, a means for acquiring temperature data from a thermometer installed in the vehicle, a means for acquiring location information from a positioning device for acquiring location data, a means for detecting an abnormality by integrating the facial recognition data, the temperature data, and the location data, a means for transmitting an alarm signal based on the abnormality detection result, a means for notifying a mobile information terminal of the transmitted alarm signal, a means for acquiring emotion data using facial expression analysis technology, and a means for adjusting the content of the notification and the level of urgency based on the emotion data. This enables monitoring of the temperature inside the vehicle, detecting left-behind people, and providing appropriate notifications according to the user's emotional state.

[1334] "Facial recognition technology" is a technology that identifies a specific person from image data captured using a camera and obtains that information.

[1335] The "camera" is an image capturing device installed in a vehicle for detecting a person's face.

[1336] A "thermometer" is a measuring device that periodically measures the temperature inside a vehicle and acquires the data.

[1337] "Location data" is location information obtained from a positioning device with a GPS function, and is data that indicates the current location of a person or vehicle.

[1338] A "positioning device" is a device that has a GPS function, is carried by a person, and is used to obtain location data.

[1339] The "means for detecting abnormalities" refers to a method of analyzing facial recognition data, temperature data, and location data, and detecting abnormalities such as high temperatures or items left behind based on that data.

[1340] A "warning signal" is a signal that is sent from the server when an abnormality is detected, to alert the user.

[1341] A "personal digital assistant" is an electronic device that receives a warning signal sent from a server and notifies the user.

[1342] "Facial expression analysis technology" is a technology that analyzes facial expressions in order to recognize the emotional state of a user.

[1343] "Emotion data" is data relating to the user's emotional state obtained using facial expression analysis technology.

[1344] The present invention provides a system that ensures safety in vehicles and prevents accidents, particularly those involving young children being left behind in vehicles, by integrating facial recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[1345] Hardware and software used

[1346] 1. Camera with facial recognition technology

[1347] Example: Face recognition algorithm using OpenCV.

[1348] Processing: Cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle. The recognition data is sent to a server.

[1349] 2. Vehicle interior temperature gauge

[1350] Example: DHT22 sensor.

[1351] Processing: Periodically measure the temperature inside the car and send the data to a server.

[1352] 3. GPS-enabled name badges

[1353] Example: Garmin GPS units.

[1354] Process: The person carries the device and periodically sends location information to a server.

[1355] 4. Emotion Engine

[1356] Examples: Amazon Rekognition, Affectiva.

[1357] Processing: Installed on mobile devices, it analyzes the user's facial expressions and voice to recognize their emotional state.

[1358] 5. Server

[1359] Examples: AWS, Google Cloud.

[1360] Processing: Detects abnormalities through integrated analysis of facial recognition data, temperature data, GPS data, and emotion data. If an abnormality is detected, an alarm signal is sent and a notification is sent to the mobile information terminal.

[1361] 6. Mobile Devices

[1362] Example: iPhone, Android device.

[1363] Processing: Receives a warning signal from the server, recognizes the user's emotions using the emotion engine, and adaptively adjusts the notification content.

[1364] System operation example

[1365] Example 1: Detecting abnormal temperatures inside a vehicle

[1366] The interior thermometer periodically measures the temperature and sends the results to the server.

[1367] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1368] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send out a warning signal.

[1369] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1370] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1371] Example 2: Detecting objects left behind in a vehicle

[1372] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1373] The server analyzes the name tag's location information to see if it is inside the vehicle, and if the location information remains inside the vehicle after 10 minutes, it determines that there is an abnormality.

[1374] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[1375] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1376] The user checks the notification to see if anyone is left in the car.

[1377] Example 3: Checking the number of people entering and leaving

[1378] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1379] The server records the facial recognition data received at the time of boarding in a database.

[1380] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[1381] The server compares the data at the time of boarding with the data at the time of disembarking, and if there is a difference in the number of people, it determines that there is an abnormality.

[1382] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[1383] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1384] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1385] Example prompts (for generative AI models)

[1386] I'm researching an in-vehicle safety system that combines facial recognition technology, thermometers, GPS functionality, an emotion engine, and mobile devices. I'd like to know more about the process and other related technologies for the following specific processing steps:

[1387] Step 1: Acquiring and sending facial recognition data

[1388] In this way, the system combines various sensors and technologies to monitor in-vehicle safety at a high level and respond quickly and appropriately in the event of an abnormality.

[1389] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1390] Step 1: Acquire and send facial recognition data

[1391] The server receives facial recognition data from the in-car camera. The in-car camera is equipped with facial recognition technology using OpenCV to detect people getting in and out of the vehicle. This data is compiled as identification information by using a facial recognition algorithm to extract feature points from facial images captured by the camera. The server then receives this data and records it in a database. The input of the facial recognition data is the facial image captured by the camera, and the output is the facial identification information sent to the server.

[1392] Step 2: Acquire and analyze temperature data

[1393] The server receives temperature data from a thermometer installed inside the vehicle. A DHT22 sensor is used as this thermometer. The temperature sensor periodically measures the temperature inside the vehicle and generates temperature data. This data is analyzed in real time by the server. During analysis, the server monitors the measured temperature data and switches to high temperature warning mode if it exceeds a certain threshold (for example, 40 degrees). The input temperature data is the temperature information sent from the DHT22 sensor, and the output is the high temperature warning judgment result made by the server.

[1394] Step 3: Acquire and verify GPS data

[1395] The server periodically obtains location information from a name tag equipped with GPS functionality. It is assumed that a Garmin GPS unit is used. After the engine is turned off, the server continues to obtain GPS data for 10 minutes. This data is compared with the vehicle's GPS data to check whether the name tag is located inside the vehicle. If the GPS name tag is still inside the vehicle 10 minutes after the engine is turned off, it is determined to be an abnormality. The GPS data input is the location information obtained from the Garmin GPS unit, and the output is the abnormality detection result by the server.

[1396] Step 4: Obtaining and sending emotion data

[1397] The device uses an emotion engine to analyze the user's emotions. Technologies such as Amazon Rekognition and Affectiva are expected to be used. The user's facial expressions and voice are captured through the mobile device's camera and microphone, and an emotion analysis algorithm analyzes them to generate emotion data. This data is sent to a server and used to adjust the urgency and content of notifications. The emotion data input is the facial and voice information captured by the mobile device's camera and microphone, and the output is data sent to the server as the emotion analysis results.

[1398] Step 5: Detect anomalies and send warning signals

[1399] The server integrates and analyzes facial recognition data, temperature data, GPS data, and emotion data, thereby detecting abnormal conditions (high temperatures, passengers left behind, mismatch in passenger numbers) in real time. If an abnormality is detected, the server immediately sends out a warning signal and notifies the mobile information terminal. The inputs for data integration and analysis are facial recognition data, temperature data, GPS data, and emotion data, and the output is the abnormality detection result and a warning signal.

[1400] Step 6: Notify and respond to users

[1401] The device receives the transmitted warning signal and notifies the user. At this time, the emotion engine recognizes the user's emotions and adjusts the content and urgency of the notification. If the user shows signs of stress or anxiety, the notification is displayed with a higher level of urgency. The user checks the content of the notification and takes appropriate action (e.g., improving the environment inside the car or rescuing anyone stranded). The input for the notification and response is the warning signal sent from the server and the user's emotional data, and the output is the action taken by the user.

[1402] (Application example 2)

[1403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1404] Conventional in-vehicle safety monitoring systems are limited to checking the number of passengers getting on and off and monitoring the temperature inside the vehicle, making it difficult to prevent accidents caused by human error or sudden temperature changes. Furthermore, they are unable to completely prevent accidents, particularly those involving young children or the elderly, from being left behind inside the vehicle. Furthermore, alert notifications do not take into account the emotional state of passengers, which can lead to delayed appropriate responses in emergencies.

[1405] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed in the vehicle, means for acquiring location information from a name-signing device equipped with a GPS function, means for recognizing people's emotions by analyzing their voices and facial expressions, means for detecting abnormalities by integrating the facial recognition data, temperature data, GPS data, and emotion data, means for issuing an alert based on the abnormality detection results, and means for notifying a computer terminal of the issued alert. This enables comprehensive monitoring of vehicle safety and prevents accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on passengers' emotional states enable immediate response even in emergencies.

[1406] "Facial recognition technology" is a technology that uses a camera to detect a person's face and identify that face as a specific individual.

[1407] A "thermometer" is a device that periodically measures the ambient temperature inside a vehicle and provides that data.

[1408] "GPS function" is a technology that uses a satellite positioning system to obtain location information on Earth.

[1409] A "name-registering device" is a device that is carried by a person and has the function of periodically transmitting location information.

[1410] "Means for recognizing a person's emotions by analyzing voice and facial expressions" refers to a technology that analyzes changes in a person's voice tone and facial expressions from audio data and video data to identify that person's emotional state.

[1411] "Means for detecting anomalies" refers to technology that integrates acquired facial recognition data, temperature data, GPS data, and emotional data, and uses the analysis results to detect unexpected events or dangerous situations.

[1412] "Means for sending alerts" refers to technology that sends information about an abnormality as audio, video, or a message when it is detected.

[1413] A "computer terminal" is an electronic device, such as a server or mobile device, that receives, processes, and displays data.

[1414] The present invention provides a system for ensuring safety in vehicles, particularly for monitoring the safety of young children and the elderly. The system operates by linking face recognition technology, a thermometer, a GPS function, an emotion analysis engine, and a computer terminal. Specific embodiments of the present invention are described below.

[1415] System configuration

[1416] 1. Camera with facial recognition technology

[1417] The cameras installed inside the vehicle recognize the faces of people getting on and off the vehicle and count the number of people. The facial recognition data is sent to a server for person detection and identification.

[1418] 2. Vehicle interior temperature gauge

[1419] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, which analyzes the temperature data and issues a warning if it detects an abnormally high temperature.

[1420] 3. Name-registering device with GPS function

[1421] The name registration device is carried by the person and periodically transmits its location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether the person remains in the vehicle after the engine has been turned off.

[1422] 4. Sentiment Analysis Engine

[1423] The emotion analysis engine is installed in the computer terminal and analyzes the user's voice and facial expressions to recognize emotions, and adjusts the alert content and urgency of notifications based on the recognized emotions.

[1424] 5. Server

[1425] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[1426] 6. Computer terminals

[1427] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. An emotion analysis engine is used to recognize the user's emotions and adaptively adjust the notification content. The user can then check the notification content and take necessary measures.

[1428] Hardware and software used

[1429] Camera: Equipped with a facial recognition camera

[1430] Thermometer: Vehicle interior thermometer

[1431] Named device: A device with GPS functionality

[1432] Server: Flask (framework), SQLAlchemy (object-relational mapping), SQLite (database)

[1433] Detection libraries: OpenCV (image processing), DeepFace (emotion analysis)

[1434] Computer terminals: smartphones, in-vehicle displays, etc.

[1435] Data processing and calculation

[1436] The server uses facial recognition technology to count the number of people inside the vehicle, encodes the data, and transmits it. Temperature and GPS data are collected in real time to monitor for abnormal conditions. An emotion analysis engine analyzes audio and video data to understand the user's emotional state. This data is integrated to detect abnormalities and send alerts at the appropriate time.

[1437] Prompt Sentence Examples

[1438] 1. Example prompt for facial recognition and emotion analysis:

[1439] "Please provide us with camera images. We will perform facial recognition and emotion analysis."

[1440] 2. Temperature data transmission prompt example:

[1441] Please enter the current interior temperature.

[1442] 3. Example of prompt for sending GPS data:

[1443] Please enter your current GPS location.

[1444] The above is a specific description of the embodiment of the invention. This system comprehensively monitors safety inside a vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, alert notifications are sent based on the emotional state of passengers, enabling a rapid response in emergencies. This significantly improves safety inside the vehicle and the user's sense of security.

[1445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1446] Step 1:

[1447] A camera equipped with facial recognition technology captures the faces of people inside the vehicle. The input is the camera image, and the output is facial recognition data. This facial recognition data extracts facial feature points and identifies specific individuals. The server receives this facial recognition data and counts the number of people getting on and off the vehicle.

[1448] Step 2:

[1449] A thermometer installed inside the vehicle periodically collects temperature data. The input is the temperature data obtained from the sensor, and the output is a structured temperature value. The server receives this temperature data and monitors for abnormal temperatures.

[1450] Step 3:

[1451] The GPS-equipped device periodically transmits location information. The input is a GPS signal, and the output is latitude and longitude information. The server receives this location information and monitors the current location of the vehicle and the location of the passengers.

[1452] Step 4:

[1453] The server integrates facial recognition data, temperature data, and GPS data to detect anomalies. The input is this integrated data, and the output is the anomaly detection results. The server performs data analysis and detects abnormal conditions (e.g., high temperature or passengers left behind).

[1454] Step 5:

[1455] Voice and facial expressions are analyzed to recognize a person's emotions. A computer terminal analyzes the user's facial expressions and voice to generate emotion analysis data. The input is audio and video data, and the output is the identified emotional state. The server also integrates this emotion data to determine the urgency of the alert.

[1456] Step 6:

[1457] The server issues an alert based on the anomaly detection results. The input is the anomaly detection results and emotion data, and the output is an alert message. The server generates and sends an appropriate alert message according to the level of urgency.

[1458] Step 7:

[1459] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. The input is an alert message, and the output is a notification display. The terminal displays a notification to the user according to the urgency level, and the user takes appropriate measures based on this.

[1460] The above is a specific processing flow of the system of the present invention, clearly showing the input, data processing, and output at each step. This system comprehensively monitors the safety of the vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on the user's emotional state enable rapid response in emergencies.

[1461] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1462] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1463] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1464] [Fourth embodiment]

[1465] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1466] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1467] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1468] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1469] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1470] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1471] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1472] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1473] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1474] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1475] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1476] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1477] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1478] The present invention provides a system designed to ensure vehicle safety and prevent accidents involving young children being left behind in vehicles. The system utilizes face recognition technology, a thermometer, a GPS function, and a mobile device.

[1479] System configuration

[1480] 1. Camera with facial recognition technology

[1481] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[1482] 2. Vehicle interior temperature gauge

[1483] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[1484] 3. GPS-enabled name badges

[1485] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[1486] 4. Server

[1487] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[1488] 5. Mobile devices

[1489] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[1490] System operation example

[1491] Example 1: Detecting abnormal temperatures inside a vehicle

[1492] The car's interior thermometer periodically measures the temperature and sends the measurement results to the server.

[1493] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[1494] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[1495] The mobile device receives the alert and displays it to the user.

[1496] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car, immediately evacuating anyone in the car, etc.).

[1497] Example 2: Detecting objects left behind in a vehicle

[1498] After the engine is shut off, the server monitors the location of the GPS name tag for 10 minutes.

[1499] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[1500] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[1501] The mobile device receives the notification and displays it to the user.

[1502] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[1503] Example 3: Checking the number of people entering and leaving

[1504] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[1505] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[1506] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[1507] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[1508] The mobile device receives the notification and displays it to the user.

[1509] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1510] With this configuration and processing, the system can effectively monitor for dangers such as people being left behind in the vehicle or high temperatures, and respond quickly to prevent accidents before they occur.

[1511] The processing flow will be explained below.

[1512] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[1513] Step 1:

[1514] The interior thermometer periodically measures the temperature and sends the results to the server.

[1515] Step 2:

[1516] The server records the received temperature data in a database.

[1517] Step 3:

[1518] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1519] Step 4:

[1520] If the temperature remains above 40 degrees for 10 minutes, the server will determine that there is an abnormality.

[1521] Step 5:

[1522] If the server detects an abnormality, it will send an alert to the mobile device.

[1523] Step 6:

[1524] The mobile terminal receives the alert and displays a notification message to the user.

[1525] Step 7:

[1526] The user checks the notification on their mobile device and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1527] Example 2: Processing flow for detecting objects left behind in a vehicle

[1528] Step 1:

[1529] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1530] Step 2:

[1531] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1532] Step 3:

[1533] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1534] Step 4:

[1535] If the server detects an abnormality, it will send an alert to the mobile device.

[1536] Step 5:

[1537] The mobile terminal receives the alert and displays a notification message to the user.

[1538] Step 6:

[1539] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[1540] Example 3: Processing flow for checking the number of people entering and leaving

[1541] Step 1:

[1542] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1543] Step 2:

[1544] The server records the facial recognition data received at the time of boarding in a database.

[1545] Step 3:

[1546] When passengers disembark, the camera again uses facial recognition technology to count the number of passengers disembarking and sends the data to a server.

[1547] Step 4:

[1548] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[1549] Step 5:

[1550] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1551] Step 6:

[1552] If the server detects an abnormality, it will send an alert to the mobile device.

[1553] Step 7:

[1554] The mobile terminal receives the alert and displays a notification message to the user.

[1555] Step 8:

[1556] The user checks the notification on their mobile device to see if anyone is left behind in the car.

[1557] As described above, a system is constructed that ensures safety inside the vehicle by performing specific operations at each processing step.

[1558] Example 1

[1559] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1560] In recent years, the number of accidents inside vehicles has been increasing. In particular, serious accidents have occurred in which young children left behind inside vehicles suffer from heatstroke due to high temperatures. Such accidents occur because the situation inside the vehicle cannot be properly monitored even after the vehicle engine has been shut off. Another problem is the ineffectiveness of checking the number of passengers and early detection of abnormalities. Therefore, there is a need for effective monitoring of dangers such as people being left behind inside vehicles and high temperatures, and for rapid response.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1562] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from a name tag equipped with a GPS function, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for automatically starting the system when the vehicle engine is started. This makes it possible to effectively monitor for dangers such as people being left behind in the vehicle or high temperatures and to respond quickly.

[1563] "Facial recognition technology" is a technology that uses a camera to recognize a person's face and uses that data to identify individual people.

[1564] "Camera" means a device that captures images or videos and, if equipped with facial recognition technology, has the ability to detect and recognize a person's face.

[1565] An "in-vehicle thermometer" is a device that measures the temperature inside a vehicle and has the function of transmitting the measured data to a server.

[1566] A "name tag with GPS functionality" is a name tag equipped with GPS technology for acquiring location information and has the function of transmitting that location data to a server.

[1567] A "server" is a computer system that receives and analyzes various data, performs the necessary processing, and outputs the results.

[1568] The "abnormality detection means" has the function of analyzing facial recognition data, temperature data, and GPS data, and detecting an abnormality if the data exceeds a set standard.

[1569] The "alert issuing means" has a function of issuing a signal to notify an abnormality when it is detected.

[1570] A "mobile terminal" is a portable electronic device that has the function of receiving alerts from a server and notifying the user.

[1571] The "means for automatically starting the system when the vehicle engine is started" has the function of automatically activating the entire linked system when the vehicle engine is started.

[1572] The following describes an embodiment of the present invention. The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. This system works by linking face recognition technology, an in-vehicle thermometer, GPS functionality, and a mobile device.

[1573] System configuration

[1574] 1. Camera with facial recognition technology

[1575] Cameras will be installed inside the vehicles and will count people entering and exiting the vehicle using facial recognition technology. Facial recognition data will be used to identify people inside the vehicle and record the number of people entering and exiting the vehicle.

[1576] The server receives and analyzes the facial recognition data sent from the camera. For example, if a child and a parent are riding in the car, the data "child = 1, parent = 1" is sent to the server.

[1577] 2. Vehicle interior temperature gauge

[1578] The interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, allowing the vehicle's interior temperature to be monitored and a warning issued if a high temperature condition occurs.

[1579] The server analyzes the temperature data and switches to alert mode if the temperature exceeds a certain threshold (for example, 40 degrees). If the high temperature continues, the server recognizes it as an abnormality and sends out an alert.

[1580] 3. GPS-enabled name badges

[1581] The name tag periodically transmits the wearer's location to a server, which then compares the name tag's location data with the vehicle's GPS data to determine if anyone remains inside the vehicle.

[1582] The server detects the possibility of a name tag being left behind inside the vehicle, especially after the engine has been shut off, and sends out an alert.

[1583] 4. Server Roles

[1584] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert to the mobile device and records it in a database.

[1585] 5. Mobile devices

[1586] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user checks the notification and takes appropriate measures.

[1587] Example of operation

[1588] Example 1: Detecting abnormal temperatures inside a vehicle

[1589] The car's interior thermometer periodically measures the temperature and sends it to the server.

[1590] The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees.

[1591] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send an alert to the mobile device.

[1592] The mobile device receives the alert and displays it to the user.

[1593] The user checks the notification and takes appropriate action (e.g., turning on the air conditioning in the car or evacuating people in the car).

[1594] Example 2: Detecting objects left behind in a vehicle

[1595] After the engine is shut off, the server monitors the GPS name tag's location information for 10 minutes.

[1596] The server analyzes the GPS data and detects an abnormality if the name tag is left inside the vehicle.

[1597] The server sends an alert to the mobile device, notifying it of the possibility of being left behind.

[1598] The mobile device receives the notification and displays it to the user.

[1599] The user checks the notification and quickly checks to see if anyone is left behind in the vehicle.

[1600] Example 3: Checking the number of people entering and leaving

[1601] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and transmit the count to a server.

[1602] When passengers disembark, the number of people is counted again using facial recognition and the data is sent to the server.

[1603] The server compares the number of passengers boarding and alighting to see if there are any discrepancies.

[1604] If there is a difference, the server detects the anomaly and sends an alert to the mobile device.

[1605] The mobile device receives the notification and displays it to the user.

[1606] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1607] Prompt Sentence Examples

[1608] Example of an input prompt for a generative AI model:

[1609] Please provide a detailed explanation of each processing step of the in-vehicle safety system, including specific operations from starting the vehicle engine to notifying the user after an abnormality is detected.

[1610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1611] Step 1: Boot the system

[1612] Input: Vehicle engine start signal

[1613] Specific operation: When the vehicle's engine starts, the server receives a start signal, which is sent from the vehicle's computer system to the server.

[1614] Output: The whole system starts up. The server checks the status of all sensors and cameras and logs their startup status.

[1615] Step 2: Facial recognition upon boarding

[1616] Input: Image data of the interior of the car captured by the camera

[1617] How it works: The camera captures real-time images of people entering the vehicle and uses a facial recognition algorithm to identify each person.

[1618] Data processing: The captured image data is analyzed using facial recognition technology and converted into person data.

[1619] Output: The person data is sent to the server, and the number of passengers and individual person information is recorded. For example, data such as "child = 1, parent = 1" is recorded.

[1620] Step 3: Regularly measure the interior temperature

[1621] Input: Temperature data sent from the car's interior thermometer

[1622] Specific operation: The car thermometer periodically measures the temperature inside the car and sends the data to the server, for example, every 5 minutes.

[1623] Data calculation: After receiving the temperature data, the server analyzes it and determines whether the temperature exceeds a certain threshold (e.g., 40 degrees).

[1624] Output: The analysis results are output, and if the temperature is abnormal, the system switches to alert mode. If the high temperature continues for 10 minutes, it is recognized as an abnormality and preparations begin to send an alert.

[1625] Step 4: Collect GPS name badge location data

[1626] Input: Location data sent from GPS name tag

[1627] Specific operation: The GPS name tag periodically transmits the location information of the person carrying it to a server.

[1628] Data processing: The server receives the nameplate location data and compares it with the GPS data inside the vehicle.

[1629] Output: After the engine is stopped, if the name tag is located inside the vehicle, the server detects the possibility that the person has been left behind and prepares to send an alert.

[1630] Step 5: Detect anomalies and send alerts

[1631] Input: Facial recognition data, temperature data, GPS data

[1632] Specific operation: The server integrates and analyzes the above data. If an abnormal condition is detected, an alert is sent immediately.

[1633] Data calculation: By integrating and analyzing each piece of data, an abnormality is detected if it exceeds set standards (for example, discrepancies in the number of passengers boarding and disembarking using facial recognition, high temperatures, or GPS data being left behind).

[1634] Output: If an anomaly is detected, the server sends an alert to the mobile device and also records it in the database.

[1635] Step 6: Notify users

[1636] Input: Alerts originating from the server

[1637] Specific behavior: The mobile device receives an alert from the server and displays a notification to the user, including the details of the anomaly and recommended actions.

[1638] Output: The user checks the notification and takes appropriate action. For example, if a high temperature warning is displayed, the user returns to the car and turns on the air conditioner, or evacuates people from the car.

[1639] These steps allow the system to ensure the safety of people inside the vehicle and respond quickly if something goes wrong.

[1640] (Application example 1)

[1641] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1642] Currently, there has been an increase in accidents involving people being left behind in autonomous vehicles, particularly young children and the elderly. The danger of abnormally high interior temperatures has also been pointed out, and these problems need to be resolved quickly and effectively. However, conventional systems lack the means to quickly detect these abnormalities and issue appropriate alerts. In light of this situation, the present invention aims to ensure the safety of people in autonomous vehicles and prevent them from being left behind or experiencing abnormally high temperatures.

[1643] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1644] In this invention, the server includes means for counting the number of people inside the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed inside the vehicle, means for acquiring location information from name tags equipped with GPS functionality, means for detecting abnormalities by integrating the facial recognition data, temperature data, and GPS data, means for issuing an alert based on the abnormality detection result, means for notifying a mobile terminal of the issued alert, and means for managing boarding and alighting in cooperation with the control system of the autonomous vehicle. This makes it possible to monitor the safety of people inside the autonomous vehicle with high accuracy, and to immediately issue an alert and prompt appropriate action if an abnormality occurs.

[1645] "Facial recognition technology" is a technology that identifies and identifies a person's face from image data acquired by a camera.

[1646] A "camera" is a device installed inside a vehicle that uses facial recognition technology to capture image data of people getting on and off the vehicle.

[1647] "Counting people" refers to the operation of measuring the number of people present in the vehicle and sending that data to the server.

[1648] A "thermometer" is a device that measures the temperature inside a vehicle and sends the data to a server.

[1649] "Temperature data" is information about the temperature inside the vehicle measured by a thermometer.

[1650] The "GPS function" is a technology for obtaining geographical location information and is installed in the name tag.

[1651] A "name tag" is a portable device equipped with GPS functionality.

[1652] "Location Information" means geographic location data obtained using GPS functionality.

[1653] "Data integration" refers to the process of combining and analyzing facial recognition data, temperature data, and GPS data to determine the overall situation.

[1654] "Detecting anomalies" means analyzing the integrated data and discovering phenomena that deviate from normal conditions.

[1655] "Sending an alert" means sending a warning message based on a detected anomaly.

[1656] A "mobile terminal" is a portable device that receives alerts from the server and notifies the user.

[1657] An "autonomous vehicle" is a vehicle whose driving is automated by a system.

[1658] A "control system" is a computer system for managing and controlling the operation of a vehicle.

[1659] "Boarding and alighting management" is the process of managing the number of people boarding and alighting a vehicle using facial recognition technology.

[1660] The present invention is a system for ensuring safety inside a vehicle and preventing accidents, particularly those involving young children and the elderly, from occurring. The system operates by linking together facial recognition technology, a thermometer, a GPS function, a mobile device, and the control system of an autonomous vehicle.

[1661] System configuration

[1662] 1. Camera with facial recognition technology

[1663] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data obtained from the cameras is sent to a server.

[1664] 2. Vehicle interior temperature gauge

[1665] A thermometer installed inside the vehicle periodically measures the temperature inside the vehicle and transmits the data to a server.

[1666] 3. GPS-enabled name badges

[1667] The name tag is carried by the person and periodically transmits location information to a server, which receives this information and compares it with the vehicle's GPS data.

[1668] 4. Server

[1669] The server integrates facial recognition data, temperature data, and GPS data to detect abnormalities, and if an abnormality is detected, the server sends an alert and records it in a database.

[1670] 5. Mobile devices

[1671] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The user can then check the notification and take the necessary measures.

[1672] 6. Autonomous Vehicle Control Systems

[1673] The control system for autonomous vehicles receives instructions from the server to manage boarding and disembarking and respond to abnormal situations.

[1674] System operation example

[1675] Example 1: Detecting abnormal temperatures inside a vehicle

[1676] The thermometer periodically measures the temperature and sends the data to a server. The server analyzes the temperature data and goes into alert mode if the temperature exceeds 40 degrees. If the high temperature continues for 10 minutes, the server determines that an abnormality has occurred and sends an alert to the mobile device. The user receives the alert and can take action, such as turning on the air conditioning in the car.

[1677] Example 2: Detecting objects left behind in a vehicle

[1678] After the engine is turned off, the server monitors the location information of the GPS name tag for 10 minutes. If the location information is inside the vehicle, it detects an abnormality and sends an alert to the mobile device. The user receives a notification and can quickly check whether anyone has been left inside the vehicle.

[1679] Example 3: Checking the number of people entering and leaving

[1680] When passengers board the vehicle, the camera uses facial recognition technology to count the number of passengers and sends the data to a server. When passengers exit the vehicle, the count is repeated and sent to the server. The server compares the number of passengers boarding and exiting the vehicle, and if there is a discrepancy, an anomaly is detected and an alert is sent to the mobile device.

[1681] Prompt Sentence Examples

[1682] Example: "Facial recognition technology was used in self-driving vehicles to prevent children from being left behind and to instantly notify mobile devices if temperatures were abnormally high. This system also utilized thermometer and GPS data for more effective monitoring. Consider implementing a similar safety monitoring system in your self-driving fleet."

[1683] Example prompt: "Give me an example of a design for a system to prevent people from being left behind in an autonomous vehicle. Explain specifically how the system combines facial recognition technology, an in-car thermometer, and GPS functionality."

[1684] This allows the user to grasp the current situation inside the vehicle in real time and take prompt action, significantly improving the safety of self-driving vehicles.

[1685] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1686] Step 1:

[1687] People counting using facial recognition

[1688] The server receives the video data sent from the camera. Then, using facial recognition technology, the server detects people's faces from this video data and counts the number of people. This data is recorded both when passengers board and when they exit the vehicle. The input is the video data, and the output is the counted number of people data. Specifically, the camera sends video of the interior of the vehicle in real time, which the server analyzes to accurately calculate the number of people inside the vehicle.

[1689] Step 2:

[1690] Acquiring temperature data

[1691] The server periodically receives temperature data from a thermometer installed inside the vehicle. It analyzes this temperature data and monitors the environmental conditions inside the vehicle. The input is the temperature data from the thermometer, and the output is the analyzed temperature information inside the vehicle. Specifically, the thermometer sends temperature data at regular intervals to the server, which then stores and analyzes the data.

[1692] Step 3:

[1693] Acquiring GPS data

[1694] The server periodically receives location information from the GPS-equipped name tag. This location information is matched with the vehicle's GPS data to confirm whether a person is present in the vehicle. The input is the GPS data from the name tag, and the output is the person's location information. Specifically, the name tag periodically transmits its location information to the server, which records and analyzes the data.

[1695] Step 4:

[1696] Data integration and analysis

[1697] The server integrates and analyzes facial recognition data, temperature data, and GPS data. It detects abnormalities based on this integrated data. The input is data from various sensors (facial recognition, temperature, GPS), and the output is the anomaly detection results. Specifically, the server comprehensively analyzes this data and detects abnormal conditions inside the vehicle.

[1698] Step 5:

[1699] Anomaly detection

[1700] The server analyzes the integrated data and determines whether an abnormality has occurred. For example, an abnormality is detected if the temperature inside the vehicle exceeds 40 degrees Celsius or if the location information of the name tag remains inside the vehicle after the engine is turned off. The input is the integrated data, and the output is the abnormality detection result and the issuance of an alert. Specifically, the server monitors the temperature data and GPS data, and detects an abnormality when certain conditions are met.

[1701] Step 6:

[1702] Sending alerts

[1703] If an anomaly is detected, the server generates an alert and sends it to the mobile device. The input is the anomaly detection result, and the output is the alert message. Specifically, when the server detects an anomaly, it immediately generates an alert message and sends it to the mobile device.

[1704] Step 7:

[1705] Mobile device notifications

[1706] The terminal receives an alert from the server and notifies the user in real time that an abnormality has occurred. The input is the alert message, and the output is a notification to the user. Specifically, when the terminal receives an alert, a pop-up notification or a voice notification is displayed to the user, urging them to take immediate action.

[1707] Step 8:

[1708] User response

[1709] The user checks the alert from the mobile device and takes the necessary action. For example, they turn on the air conditioner in the car or check the interior of the car. The input is the alert notification, and the output is the actual response action. Specifically, the user checks the alert and quickly operates the air conditioner or checks the interior of the car to ensure safety.

[1710] The above processing steps enable safety monitoring inside autonomous vehicles. By combining multiple technologies such as facial recognition, GPS, and temperature sensors, this system enhances safety inside the vehicle and is able to quickly detect and respond to abnormalities.

[1711] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1712] The present invention is a system designed to ensure safety in vehicles, particularly to prevent accidents involving young children being left behind in vehicles. The system works in conjunction with face recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[1713] System configuration

[1714] 1. Camera with facial recognition technology

[1715] The cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle, and the facial recognition data is sent to a server for person detection and identification.

[1716] 2. Vehicle interior temperature gauge

[1717] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server. The temperature data is used to monitor the environmental conditions inside the vehicle and issue warnings if an abnormally high temperature occurs.

[1718] 3. GPS-enabled name badges

[1719] The name tag is carried by the person and periodically transmits location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether children or other people are left in the car after the engine is turned off.

[1720] 4. Emotion Engine

[1721] The emotion engine is installed on mobile devices and analyzes the user's facial expressions and voice to recognize their emotions, adjusting the alert content and urgency of notifications based on the recognized emotions.

[1722] 5. Server

[1723] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[1724] 6. Mobile Devices

[1725] The mobile device receives an alert from the server and notifies the user that an abnormality has occurred. The emotion engine recognizes the user's emotions and adaptively adjusts the notification content. The user can check the notification content and take necessary measures.

[1726] System operation example

[1727] Example 1: Detecting abnormal temperatures inside a vehicle

[1728] The interior thermometer periodically measures the temperature and sends the results to the server.

[1729] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1730] If the server remains in a high temperature state for 10 minutes, it will be determined to be abnormal.

[1731] If the server detects an abnormality, it will send an alert to the mobile device.

[1732] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user is showing signs of stress or anxiety, the notification will be displayed with a higher level of urgency.

[1733] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1734] Example 2: Detecting objects left behind in a vehicle

[1735] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1736] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1737] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1738] If the server detects an abnormality, it will send an alert to the mobile device.

[1739] The mobile device receives the alert and uses an emotion engine to recognize the user's emotions. If the user shows signs of impatience or nervousness, the notification will be displayed with a higher level of urgency.

[1740] The user checks the notification to see if anyone is left in the car.

[1741] Example 3: Checking the number of people entering and leaving

[1742] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1743] The server records the facial recognition data received at the time of boarding in a database.

[1744] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[1745] The server compares the data at the time of boarding with the data at the time of disembarking to check whether there are any discrepancies in the number of people.

[1746] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1747] If the server detects an abnormality, it will send an alert to the mobile device.

[1748] The mobile device receives the alert and uses an emotion engine to recognize the user's emotion. If the user is calm, a normal notification is displayed. If the user is upset, the notification is displayed with a higher level of urgency.

[1749] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1750] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[1751] The processing flow will be explained below.

[1752] Example 1: Processing flow for detecting abnormal temperature inside a vehicle

[1753] Step 1:

[1754] The interior thermometer periodically measures the temperature and sends the results to the server.

[1755] Step 2:

[1756] The server records the received temperature data in a database.

[1757] Step 3:

[1758] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1759] Step 4:

[1760] The server will determine that an abnormality has occurred if the temperature remains above 40 degrees for 10 minutes.

[1761] Step 5:

[1762] If the server detects an abnormality, it will send an alert to the mobile device.

[1763] Step 6:

[1764] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1765] Step 7:

[1766] The emotion engine analyzes the received data and determines whether the user is exhibiting stress or anxiety.

[1767] Step 8:

[1768] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1769] Step 9:

[1770] The user will check the notification and take appropriate action, such as improving the in-car environment or conducting rescue operations.

[1771] Example 2: Processing flow for detecting objects left behind in a vehicle

[1772] Step 1:

[1773] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1774] Step 2:

[1775] The server analyzes the location information of the name tag to see if it is inside the vehicle.

[1776] Step 3:

[1777] If the GPS name tag's location information is still inside the vehicle 10 minutes after the engine is turned off, the server will determine that there is an abnormality.

[1778] Step 4:

[1779] If the server detects an abnormality, it will send an alert to the mobile device.

[1780] Step 5:

[1781] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1782] Step 6:

[1783] The emotion engine analyzes the received data and determines whether the user is expressing impatience or tension.

[1784] Step 7:

[1785] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1786] Step 8:

[1787] The user will check the notification and act quickly to check if anyone is left in the vehicle.

[1788] Example 3: Processing flow for checking the number of people entering and leaving

[1789] Step 1:

[1790] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1791] Step 2:

[1792] The server records the facial recognition data received at the time of boarding in a database.

[1793] Step 3:

[1794] When passengers exit the vehicle, the onboard camera again uses facial recognition technology to count the number of people exiting the vehicle and sends the data to a server.

[1795] Step 4:

[1796] The server compares the number of passengers boarding and alighting to check whether there is a discrepancy in the numbers.

[1797] Step 5:

[1798] If there is a discrepancy in the number of people, the server determines that there is an abnormality.

[1799] Step 6:

[1800] If the server detects an abnormality, it will send an alert to the mobile device.

[1801] Step 7:

[1802] The mobile device receives the alert and activates an emotion engine to analyze the user's facial expressions and voice using the user's camera and microphone.

[1803] Step 8:

[1804] The emotion engine analyzes the received data and determines whether the user is calm or upset.

[1805] Step 9:

[1806] Based on the results from the emotion engine, the mobile device adjusts the urgency of the notification and displays it to the user.

[1807] Step 10:

[1808] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1809] In this way, by performing clear processing at each step, it is possible to ensure safety inside the vehicle while recognizing the emotional state of the user throughout the system.

[1810] Example 2

[1811] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1812] The purpose of this invention is to ensure safety inside vehicles, particularly to prevent accidents involving young children being left behind inside. Conventional systems have difficulty properly monitoring and managing passenger entry and exit and the interior environment, posing particular risks to children left behind and high temperatures. To address these issues, a new method is needed that integrates facial recognition technology, a thermometer, GPS functionality, an emotion engine, and mobile devices.

[1813] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, a means for acquiring temperature data from a thermometer installed in the vehicle, a means for acquiring location information from a positioning device for acquiring location data, a means for detecting an abnormality by integrating the facial recognition data, the temperature data, and the location data, a means for transmitting an alarm signal based on the abnormality detection result, a means for notifying a mobile information terminal of the transmitted alarm signal, a means for acquiring emotion data using facial expression analysis technology, and a means for adjusting the content of the notification and the level of urgency based on the emotion data. This enables monitoring of the temperature inside the vehicle, detecting left-behind people, and providing appropriate notifications according to the user's emotional state.

[1814] "Facial recognition technology" is a technology that identifies a specific person from image data captured using a camera and obtains that information.

[1815] The "camera" is an image capturing device installed in a vehicle for detecting a person's face.

[1816] A "thermometer" is a measuring device that periodically measures the temperature inside a vehicle and acquires the data.

[1817] "Location data" is location information obtained from a positioning device with a GPS function, and is data that indicates the current location of a person or vehicle.

[1818] A "positioning device" is a device that has a GPS function, is carried by a person, and is used to obtain location data.

[1819] The "means for detecting abnormalities" refers to a method of analyzing facial recognition data, temperature data, and location data, and detecting abnormalities such as high temperatures or items left behind based on that data.

[1820] A "warning signal" is a signal that is sent from the server when an abnormality is detected, to alert the user.

[1821] A "personal digital assistant" is an electronic device that receives a warning signal sent from a server and notifies the user.

[1822] "Facial expression analysis technology" is a technology that analyzes facial expressions in order to recognize the emotional state of a user.

[1823] "Emotion data" is data relating to the user's emotional state obtained using facial expression analysis technology.

[1824] The present invention provides a system that ensures safety in vehicles and prevents accidents, particularly those involving young children being left behind in vehicles, by integrating facial recognition technology, a thermometer, a GPS function, an emotion engine, and a mobile device.

[1825] Hardware and software used

[1826] 1. Camera with facial recognition technology

[1827] Example: Face recognition algorithm using OpenCV.

[1828] Processing: Cameras are installed inside the vehicles and use facial recognition technology to count people getting on and off the vehicle. The recognition data is sent to a server.

[1829] 2. Vehicle interior temperature gauge

[1830] Example: DHT22 sensor.

[1831] Processing: Periodically measure the temperature inside the car and send the data to a server.

[1832] 3. GPS-enabled name badges

[1833] Example: Garmin GPS units.

[1834] Process: The person carries the device and periodically sends location information to a server.

[1835] 4. Emotion Engine

[1836] Examples: Amazon Rekognition, Affectiva.

[1837] Processing: Installed on mobile devices, it analyzes the user's facial expressions and voice to recognize their emotional state.

[1838] 5. Server

[1839] Examples: AWS, Google Cloud.

[1840] Processing: Detects abnormalities through integrated analysis of facial recognition data, temperature data, GPS data, and emotion data. If an abnormality is detected, an alarm signal is sent and a notification is sent to the mobile information terminal.

[1841] 6. Mobile Devices

[1842] Example: iPhone, Android device.

[1843] Processing: Receives a warning signal from the server, recognizes the user's emotions using the emotion engine, and adaptively adjusts the notification content.

[1844] System operation example

[1845] Example 1: Detecting abnormal temperatures inside a vehicle

[1846] The interior thermometer periodically measures the temperature and sends the results to the server.

[1847] The server analyzes the temperature data and switches to alert mode if the measurement exceeds 40 degrees.

[1848] If the high temperature continues for 10 minutes, the server will determine that there is an abnormality and send out a warning signal.

[1849] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1850] The user checks the notification and takes appropriate action (e.g., improving the in-car environment or conducting rescue operations).

[1851] Example 2: Detecting objects left behind in a vehicle

[1852] After the engine is turned off, the server periodically obtains the location information of the GPS name tag for 10 minutes.

[1853] The server analyzes the name tag's location information to see if it is inside the vehicle, and if the location information remains inside the vehicle after 10 minutes, it determines that there is an abnormality.

[1854] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[1855] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1856] The user checks the notification to see if anyone is left in the car.

[1857] Example 3: Checking the number of people entering and leaving

[1858] When passengers board the vehicle, the onboard camera uses facial recognition technology to count the number of passengers and transmits the data to a server.

[1859] The server records the facial recognition data received at the time of boarding in a database.

[1860] When passengers disembark, the camera again uses facial recognition technology to count the number of people disembarking and sends the data to a server.

[1861] The server compares the data at the time of boarding with the data at the time of disembarking, and if there is a difference in the number of people, it determines that there is an abnormality.

[1862] If an abnormality is detected, the server sends an alert signal to notify the mobile device.

[1863] The mobile device receives the warning signal, recognizes the user's emotions using an emotion engine, and adjusts the urgency of the notification.

[1864] The user checks the notification and takes appropriate action to check the situation inside the vehicle (e.g., rescue anyone left inside the vehicle).

[1865] Example prompts (for generative AI models)

[1866] I'm researching an in-vehicle safety system that combines facial recognition technology, thermometers, GPS functionality, an emotion engine, and mobile devices. I'd like to know more about the process and other related technologies for the following specific processing steps:

[1867] Step 1: Acquiring and sending facial recognition data

[1868] In this way, the system combines various sensors and technologies to monitor in-vehicle safety at a high level and respond quickly and appropriately in the event of an abnormality.

[1869] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1870] Step 1: Acquire and send facial recognition data

[1871] The server receives facial recognition data from the in-car camera. The in-car camera is equipped with facial recognition technology using OpenCV to detect people getting in and out of the vehicle. This data is compiled as identification information by using a facial recognition algorithm to extract feature points from facial images captured by the camera. The server then receives this data and records it in a database. The input of the facial recognition data is the facial image captured by the camera, and the output is the facial identification information sent to the server.

[1872] Step 2: Acquire and analyze temperature data

[1873] The server receives temperature data from a thermometer installed inside the vehicle. A DHT22 sensor is used as this thermometer. The temperature sensor periodically measures the temperature inside the vehicle and generates temperature data. This data is analyzed in real time by the server. During analysis, the server monitors the measured temperature data and switches to high temperature warning mode if it exceeds a certain threshold (for example, 40 degrees). The input temperature data is the temperature information sent from the DHT22 sensor, and the output is the high temperature warning judgment result made by the server.

[1874] Step 3: Acquire and verify GPS data

[1875] The server periodically obtains location information from a name tag equipped with GPS functionality. It is assumed that a Garmin GPS unit is used. After the engine is turned off, the server continues to obtain GPS data for 10 minutes. This data is compared with the vehicle's GPS data to check whether the name tag is located inside the vehicle. If the GPS name tag is still inside the vehicle 10 minutes after the engine is turned off, it is determined to be an abnormality. The GPS data input is the location information obtained from the Garmin GPS unit, and the output is the abnormality detection result by the server.

[1876] Step 4: Obtaining and sending emotion data

[1877] The device uses an emotion engine to analyze the user's emotions. Technologies such as Amazon Rekognition and Affectiva are expected to be used. The user's facial expressions and voice are captured through the mobile device's camera and microphone, and an emotion analysis algorithm analyzes them to generate emotion data. This data is sent to a server and used to adjust the urgency and content of notifications. The emotion data input is the facial and voice information captured by the mobile device's camera and microphone, and the output is data sent to the server as the emotion analysis results.

[1878] Step 5: Detect anomalies and send warning signals

[1879] The server integrates and analyzes facial recognition data, temperature data, GPS data, and emotion data, thereby detecting abnormal conditions (high temperatures, passengers left behind, mismatch in passenger numbers) in real time. If an abnormality is detected, the server immediately sends out a warning signal and notifies the mobile information terminal. The inputs for data integration and analysis are facial recognition data, temperature data, GPS data, and emotion data, and the output is the abnormality detection result and a warning signal.

[1880] Step 6: Notify and respond to users

[1881] The device receives the transmitted warning signal and notifies the user. At this time, the emotion engine recognizes the user's emotions and adjusts the content and urgency of the notification. If the user shows signs of stress or anxiety, the notification is displayed with a higher level of urgency. The user checks the content of the notification and takes appropriate action (e.g., improving the environment inside the car or rescuing anyone stranded). The input for the notification and response is the warning signal sent from the server and the user's emotional data, and the output is the action taken by the user.

[1882] (Application example 2)

[1883] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1884] Conventional in-vehicle safety monitoring systems are limited to checking the number of passengers getting on and off and monitoring the temperature inside the vehicle, making it difficult to prevent accidents caused by human error or sudden temperature changes. Furthermore, they are unable to completely prevent accidents, particularly those involving young children or the elderly, from being left behind inside the vehicle. Furthermore, alert notifications do not take into account the emotional state of passengers, which can lead to delayed appropriate responses in emergencies.

[1885] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for counting the number of people in the vehicle using a camera equipped with facial recognition technology, means for acquiring temperature data from a thermometer installed in the vehicle, means for acquiring location information from a name-signing device equipped with a GPS function, means for recognizing people's emotions by analyzing their voices and facial expressions, means for detecting abnormalities by integrating the facial recognition data, temperature data, GPS data, and emotion data, means for issuing an alert based on the abnormality detection results, and means for notifying a computer terminal of the issued alert. This enables comprehensive monitoring of vehicle safety and prevents accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on passengers' emotional states enable immediate response even in emergencies.

[1886] "Facial recognition technology" is a technology that uses a camera to detect a person's face and identify that face as a specific individual.

[1887] A "thermometer" is a device that periodically measures the ambient temperature inside a vehicle and provides that data.

[1888] "GPS function" is a technology that uses a satellite positioning system to obtain location information on Earth.

[1889] A "name-registering device" is a device that is carried by a person and has the function of periodically transmitting location information.

[1890] "Means for recognizing a person's emotions by analyzing voice and facial expressions" refers to a technology that analyzes changes in a person's voice tone and facial expressions from audio data and video data to identify that person's emotional state.

[1891] "Means for detecting anomalies" refers to technology that integrates acquired facial recognition data, temperature data, GPS data, and emotional data, and uses the analysis results to detect unexpected events or dangerous situations.

[1892] "Means for sending alerts" refers to technology that sends information about an abnormality as audio, video, or a message when it is detected.

[1893] A "computer terminal" is an electronic device, such as a server or mobile device, that receives, processes, and displays data.

[1894] The present invention provides a system for ensuring safety in vehicles, particularly for monitoring the safety of young children and the elderly. The system operates by linking face recognition technology, a thermometer, a GPS function, an emotion analysis engine, and a computer terminal. Specific embodiments of the present invention are described below.

[1895] System configuration

[1896] 1. Camera with facial recognition technology

[1897] The cameras installed inside the vehicle recognize the faces of people getting on and off the vehicle and count the number of people. The facial recognition data is sent to a server for person detection and identification.

[1898] 2. Vehicle interior temperature gauge

[1899] The vehicle interior thermometer periodically measures the temperature inside the vehicle and sends the data to a server, which analyzes the temperature data and issues a warning if it detects an abnormally high temperature.

[1900] 3. Name-registering device with GPS function

[1901] The name registration device is carried by the person and periodically transmits its location information. The server receives this location information and compares it with the vehicle's GPS data. In particular, it monitors whether the person remains in the vehicle after the engine has been turned off.

[1902] 4. Sentiment Analysis Engine

[1903] The emotion analysis engine is installed in the computer terminal and analyzes the user's voice and facial expressions to recognize emotions, and adjusts the alert content and urgency of notifications based on the recognized emotions.

[1904] 5. Server

[1905] The server integrates facial recognition data, temperature data, GPS data, and emotion data to detect abnormalities. If an abnormality is detected, the server sends an alert and records it in a database.

[1906] 6. Computer terminals

[1907] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. An emotion analysis engine is used to recognize the user's emotions and adaptively adjust the notification content. The user can then check the notification content and take necessary measures.

[1908] Hardware and software used

[1909] Camera: Equipped with a facial recognition camera

[1910] Thermometer: Vehicle interior thermometer

[1911] Named device: A device with GPS functionality

[1912] Server: Flask (framework), SQLAlchemy (object-relational mapping), SQLite (database)

[1913] Detection libraries: OpenCV (image processing), DeepFace (emotion analysis)

[1914] Computer terminals: smartphones, in-vehicle displays, etc.

[1915] Data processing and calculation

[1916] The server uses facial recognition technology to count the number of people inside the vehicle, encodes the data, and transmits it. Temperature and GPS data are collected in real time to monitor for abnormal conditions. An emotion analysis engine analyzes audio and video data to understand the user's emotional state. This data is integrated to detect abnormalities and send alerts at the appropriate time.

[1917] Prompt Sentence Examples

[1918] 1. Example prompt for facial recognition and emotion analysis:

[1919] "Please provide us with camera images. We will perform facial recognition and emotion analysis."

[1920] 2. Temperature data transmission prompt example:

[1921] Please enter the current interior temperature.

[1922] 3. Example of prompt for sending GPS data:

[1923] Please enter your current GPS location.

[1924] The above is a specific description of the embodiment of the invention. This system comprehensively monitors safety inside a vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, alert notifications are sent based on the emotional state of passengers, enabling a rapid response in emergencies. This significantly improves safety inside the vehicle and the user's sense of security.

[1925] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1926] Step 1:

[1927] A camera equipped with facial recognition technology captures the faces of people inside the vehicle. The input is the camera image, and the output is facial recognition data. This facial recognition data extracts facial feature points and identifies specific individuals. The server receives this facial recognition data and counts the number of people getting on and off the vehicle.

[1928] Step 2:

[1929] A thermometer installed inside the vehicle periodically collects temperature data. The input is the temperature data obtained from the sensor, and the output is a structured temperature value. The server receives this temperature data and monitors for abnormal temperatures.

[1930] Step 3:

[1931] The GPS-equipped device periodically transmits location information. The input is a GPS signal, and the output is latitude and longitude information. The server receives this location information and monitors the current location of the vehicle and the location of the passengers.

[1932] Step 4:

[1933] The server integrates facial recognition data, temperature data, and GPS data to detect anomalies. The input is this integrated data, and the output is the anomaly detection results. The server performs data analysis and detects abnormal conditions (e.g., high temperature or passengers left behind).

[1934] Step 5:

[1935] Voice and facial expressions are analyzed to recognize a person's emotions. A computer terminal analyzes the user's facial expressions and voice to generate emotion analysis data. The input is audio and video data, and the output is the identified emotional state. The server also integrates this emotion data to determine the urgency of the alert.

[1936] Step 6:

[1937] The server issues an alert based on the anomaly detection results. The input is the anomaly detection results and emotion data, and the output is an alert message. The server generates and sends an appropriate alert message according to the level of urgency.

[1938] Step 7:

[1939] The computer terminal receives an alert from the server and notifies the user that an abnormality has occurred. The input is an alert message, and the output is a notification display. The terminal displays a notification to the user according to the urgency level, and the user takes appropriate measures based on this.

[1940] The above is a specific processing flow of the system of the present invention, clearly showing the input, data processing, and output at each step. This system comprehensively monitors the safety of the vehicle, making it possible to prevent accidents, particularly those involving young children or the elderly, from occurring. Furthermore, timely alert notifications based on the user's emotional state enable rapid response in emergencies.

[1941] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1942] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1943] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1944] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification pr...

Claims

1. 1. A system for monitoring persons in a vehicle, comprising: A method for counting the number of people inside a vehicle using a camera equipped with facial recognition technology; means for acquiring temperature data from a thermometer installed in the vehicle; A means for acquiring location information of a name tag equipped with a GPS function; a means for detecting anomalies by integrating the facial recognition data, temperature data, and GPS data; means for issuing an alert based on the anomaly detection result; and means for notifying the mobile terminal of the issued alert.

2. The system according to claim 1 , wherein the abnormality detection means issues an alert when the detected temperature data exceeds a predetermined threshold.

3. 2. The system according to claim 1, wherein the abnormality detection means issues an alert if the location information of a name tag equipped with a GPS function is present inside the vehicle within a predetermined time after the engine has been stopped.

Images