system

Abstract

Provide a system. 【Solution means】 Means for analyzing data acquired by a detector and an image sensor mounted on a vehicle to detect abnormalities, Means for notifying the detected abnormalities to the owner terminal in real time, Means for stopping the power unit of the vehicle based on the detection of abnormalities, Means for tracking the position of the vehicle using a position information system and providing the information to the owner, Communication network means for jointly implementing security measures by sharing security information with other users, Means for providing the owner with information to support safe driving based on past operation data, Means for transmitting information to the owner in real time when abnormal vibrations or impacts are detected during driving, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Automobile theft remains a serious problem, especially for high-end vehicles, where the risk of theft is high. Conventional automotive security systems have problems such as taking a long time to detect abnormalities and being difficult to track after a theft occurs. Also, it is difficult to take effective measures by coordinating multiple security systems. Furthermore, insufficient support for safe driving is also one of the problems.

Means for Solving the Problems

[0005] This invention provides a system that analyzes data in real time using sensors and cameras mounted on a vehicle and rapidly detects abnormalities. When an abnormality is detected, the system has the function of notifying the owner in real time and shutting down the engine. It also has the function of tracking the vehicle's location using GPS and providing the owner with accurate location information. Furthermore, a network function allows for the sharing of security information with other users, enabling collaborative security measures. To support safe driving, it provides advice based on past driving data, thereby improving the vehicle's theft prevention and tracking capabilities.

[0006] A "sensor" is a device that detects the state of the environment or objects and outputs the data.

[0007] A "camera" is a device that captures images and videos and records them as digital data.

[0008] "Data analysis" is the process of processing acquired information and extracting meaningful information.

[0009] Anomaly detection is a technique for discovering behavior that deviates from normal operation or state.

[0010] "Real-time notification" is a system that immediately transmits information to relevant parties when a specific event occurs.

[0011] "Engine shutdown" is a function that stops the vehicle's powertrain.

[0012] "GPS tracking" is a technology that uses the Global Positioning System to obtain location information and track the movement of an object.

[0013] "Network functionality" refers to technology that enables multiple devices and systems to cooperate and share information.

[0014] "Safe driving support" refers to a function that provides drivers with information and advice to promote safe driving. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a RAM (Random Access Memory) with a reference number is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a storage with a reference number is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The vehicle security system of the present invention is an AI-based system aimed at preventing vehicle theft and tracking, and functions through the coordinated operation of multiple components. The system consists of sensors and cameras on the vehicle itself, a central server, and the owner's terminal (such as a smartphone).

[0037] System Configuration

[0038] 1. Terminals: Sensors and cameras mounted on the vehicle monitor the surrounding environment in real time, constantly checking for any abnormalities. When these devices detect abnormal events such as doors opening and closing, windows breaking, or movement inside the vehicle, they immediately send the information to the server.

[0039] 2. Server: Upon receiving notification of an anomaly, the server analyzes the data to determine the type and severity of the anomaly. If an anomaly is confirmed, a real-time push notification is sent to the owner's smartphone. The server also uses GPS data to determine the vehicle's current location and sends this location information to the device.

[0040] 3. User: Owners can check the vehicle's status from their smartphones via the notifications they receive. The application allows them to view the vehicle's current location on a map and also provides a function to stream video from cameras installed on the vehicle in real time.

[0041] Usage example

[0042] For example, if a vehicle door is forcibly opened due to unauthorized access at night, the terminal immediately detects the anomaly and sends the information to the server. In response, the server sends a notification to the owner's smartphone based on a predetermined protocol. Furthermore, the server automatically shuts down the engine to prevent the vehicle from moving. Meanwhile, the user can open the application to check the vehicle's current location and camera footage of the surrounding area, and report to the police if necessary.

[0043] By building such a system, vehicle theft prevention and rapid tracking are possible, providing owners with peace of mind and safety. Furthermore, sharing crime prevention information with other users improves the overall level of crime prevention in the community. This system is particularly effective for luxury cars, where a high level of security is required.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The terminal keeps the vehicle's sensors and cameras running continuously to monitor things like door opening and closing, movement inside the vehicle, and glass breakage. The data obtained from the sensors is analyzed in real time, and if an abnormality beyond the normal range is detected, it is immediately communicated to the server.

[0047] Step 2:

[0048] The server receives abnormal data sent from the terminal and analyzes its contents. It determines the type and severity of the abnormality and identifies the appropriate action to take. The analysis performed in this step determines whether or not the engine shutdown trigger is pulled.

[0049] Step 3:

[0050] If an anomaly is detected by the server, it will immediately send a notification to the owner's device. The notification will include detailed information about the anomaly, the time it occurred, and even real-time location information, which the owner can view on their smartphone.

[0051] Step 4:

[0052] The server uses location services to continuously determine the vehicle's current location. This location information is kept up-to-date and updated on the user's device as needed.

[0053] Step 5:

[0054] Users can check notifications and verify the vehicle's status and current location through a smartphone application. They can also stream real-time video from the vehicle's cameras to visually confirm any suspicious situations.

[0055] Step 6:

[0056] If the malfunction is not resolved, the device will continue to emit an alarm and maintain engine shutdown. This physically prevents unauthorized engine starting or vehicle movement.

[0057] These steps ensure vehicle safety and enable prompt and appropriate responses to users.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Existing vehicle security systems have challenges in terms of rapid response after anomaly detection and secure vehicle management. Furthermore, they lack sufficient functions to prevent unauthorized access to vehicles and to provide information to owners, making it difficult to prevent damage resulting from such access. In addition, the lack of situational awareness and effective information sharing with other users results in insufficient overall security measures. There is a need to solve these problems and dramatically improve vehicle safety.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for analyzing data acquired by a sensor device and a camera device to detect anomalies, means for analyzing the data on an analysis platform and evaluating the severity of the anomaly, and means for providing video data to the owner in real time using streaming distribution technology. This enables a rapid response after anomaly detection and allows the owner to check and manage the safety of the vehicle in real time.

[0063] A "sensor device" is a device used to detect physical changes in the surrounding environment and is used to detect abnormal behavior or changes.

[0064] A "photography device" is a device used to acquire visual information and is used to understand the situation around a vehicle.

[0065] An "analysis platform" is a foundational technology for receiving, processing, and analyzing data, and is used to evaluate anomalies and determine their severity.

[0066] "Streaming technology" is a technology that continuously transmits video data in real time over the internet and is used to instantly check the situation in remote locations.

[0067] "Power system" refers to the equipment used to drive a vehicle, and mainly includes engines and motors.

[0068] "Location measurement technology" refers to technology used to determine geographical location, and generally involves using GPS or similar systems to identify the location information of a moving object.

[0069] "Communication means" refers to the media and technologies used to send and receive information, and is used for the purpose of sharing crime prevention information with others through a communication network.

[0070] "Personal identification technology" refers to technology used to distinguish a specific person from others, and is used to prevent unauthorized access using biometric information or passwords.

[0071] This system utilizes complex technologies to enhance vehicle safety, handling everything from anomaly detection to information sharing in a consistent manner. Specifically, it is implemented through the following combination of hardware and software.

[0072] The terminal includes a sensor device and a camera device mounted on the vehicle. The sensor device constantly monitors for physical anomalies, and when an anomaly is detected, the camera device acquires the relevant video footage and transmits the data to the server. For example, if a door is opened illegally, the terminal immediately transmits that information to the server.

[0073] The server operates on a cloud infrastructure such as AWS® and provides an analysis platform for analyzing received data. This platform utilizes machine learning models such as TENSORFLOW® to determine the severity of anomalies. When an anomaly is identified, Firebase Cloud Messaging is used to immediately notify the owner and, if necessary, to instruct the shutdown of the power unit. In addition, location measurement technology is used to obtain and track the vehicle's current location, and this information is communicated to the user in real time.

[0074] Users can receive notifications via their smartphones and check the vehicle's status and location using a dedicated application. This app integrates with the Google® Maps API to display the vehicle's location on a map and provides real-time video using streaming technology. If necessary, this information can be used to immediately report the incident to the police.

[0075] As a concrete example, if unauthorized access to a vehicle occurs at night, the terminal detects the anomaly and sends data to the server. The server analyzes this data and sends a push notification to the user's smartphone, while also displaying the vehicle's current location on a map. The user can then view the camera footage through the application and make a safe assessment of the situation.

[0076] An example of a prompt would be: "Please describe the specific processes involved in an AI-based vehicle security system, from anomaly detection and analysis to sending real-time notifications to the user. Focus on how TensorFlow and Firebase Cloud Messaging are utilized."

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The terminal uses sensors and cameras mounted on the vehicle to monitor the surrounding environment in real time. Inputs include capturing the opening and closing status of the vehicle doors, the state of the windows, and movement inside the vehicle. Based on the data from these sensors, it detects anomalies, and if an anomaly is determined, it immediately transmits the sensor data and video from the camera to a central server. The output at this stage specifically consists of the anomaly determination result and video data.

[0080] Step 2:

[0081] The server receives abnormal data and video data transmitted from the terminal. This data, obtained as input, is analyzed using an analysis platform. Specifically, generative AI models and machine learning frameworks are used to classify the data and determine the type and severity of the anomaly. If an anomaly is confirmed as a result of the analysis, the results are formatted as information to notify the owner, and push notification data is generated as output.

[0082] Step 3:

[0083] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's smartphone. This notification includes detailed information about the detected anomaly and the vehicle's current location. The server utilizes the Google Maps API based on the input data to calculate the vehicle's location and add it to the information sent to the owner. The output is a notification message that the owner can immediately recognize.

[0084] Step 4:

[0085] The user checks the push notification received on their smartphone. By launching the application, they can view the vehicle's location on a map in real time and watch the camera footage via streaming. Based on the information received, the user can take prompt action, such as reporting to the police, if necessary. At this stage, the input is notification data from the server, and the output is the specific action taken by the user.

[0086] Step 5:

[0087] If the server determines that an anomaly is serious, it sends a command to the vehicle to shut down the powertrain. The input is the type and severity of the anomaly determined in the previous step, and the output is a control signal that shuts down the engine. This system is designed to minimize the impact of unauthorized access.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Automobiles are an important means of transportation in daily life, but their theft and misuse are serious problems. Furthermore, systems capable of quickly detecting abnormal vibrations and shocks that occur while a vehicle is in motion and notifying the owner are still insufficient. There is also a need for a system that efficiently shares vehicle security information with other users to improve overall community safety.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for analyzing data acquired by detectors and image sensors mounted on the vehicle to detect anomalies, means for notifying the owner's terminal of the detected anomaly in real time, and means for shutting down the vehicle's power unit based on the detection of an anomaly. This enables vehicle theft prevention, early detection and appropriate response to abnormal events, and rapid sharing of security information.

[0093] A "detector" is a general term for a sensor that is installed on a vehicle and can detect physical changes or abnormalities in the environment.

[0094] An "image sensor" is a device that converts light into electrical signals and acquires surrounding visual information as digital data.

[0095] An "abnormality" refers to a condition that deviates from the normal operation or environmental conditions of a vehicle, including, for example, unauthorized access, unexpected vibrations, or shocks.

[0096] "Owner's terminal" refers to a terminal device such as a mobile phone or tablet used by the vehicle owner to receive various information.

[0097] "Real-time notification" refers to a technological means that allows the owner to be notified almost immediately as soon as an anomaly is detected.

[0098] "Stopping the power system" means forcibly stopping the vehicle's engine and other drive mechanisms to prevent the vehicle from moving illegally.

[0099] A "location information system" is a system that uses the Global Positioning System (GPS) and other technologies to determine and track the current location of a vehicle.

[0100] A "communication network means" is a system that uses network infrastructure for sending and receiving data to share crime prevention information with others.

[0101] "Past operation data" refers to recorded information regarding the vehicle's past driving history and usage.

[0102] "Information to support safe driving" refers to information provided to vehicle owners, including advice and warnings to improve driving behavior and enhance safety.

[0103] Embodiments of this invention consist of a system comprising hardware mounted on a vehicle, a terminal used by the owner, and a central server. This system acquires and analyzes data from detectors and image sensors mounted on the vehicle in real time, thereby enabling efficient security monitoring.

[0104] The server receives data transmitted from the vehicle and uses TensorFlow, AI analysis software, to detect anomalies. Firebase Cloud Messaging is used to immediately determine the type and severity of the anomaly and send a notification to the owner's device. This notification is sent via push notification to the owner's device, providing a system for immediate response to anomalies.

[0105] For example, if a detector senses abnormal vibrations caused by a collision between a vehicle and another object, the server analyzes the impact data and immediately sends a notification to the owner's device stating, "An impact has been detected on your vehicle. Please check the location and video." Furthermore, it is possible to track the vehicle's current location using a location information system and provide the owner with detailed location information.

[0106] Furthermore, by utilizing communication networks, this crime prevention information can be shared with other users, thereby improving security throughout the entire area. For example, this can raise crime prevention awareness among nearby users by notifying them that "suspicious activity has been detected."

[0107] A concrete example of this system's application is a prompt to the generative AI model such as, "Please tell me what types of sensors and cameras should be added for anomaly detection, and what algorithm the AI ​​system can use to quickly and efficiently identify anomalies." Such prompts have the potential to suggest various additional technologies and optimization directions in actual implementation.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The terminal collects various data in real time from detectors and image sensors mounted on the vehicle. Inputs include vibration data and video data, and outputs generate packets for sending this data to a central server. Specifically, it continuously monitors the surrounding environment and, if an anomaly occurs, immediately prepares to assemble it into data and send it.

[0111] Step 2:

[0112] The server receives data sent from the terminal and analyzes its contents. Input data includes vibration intensity and image frames. The server uses TensorFlow to perform analysis to determine whether there are any anomalies in this data. The output generates analysis results regarding the presence and type of anomalies. Specifically, the process involves applying an AI model to the data and calculating a confidence score for the anomalies.

[0113] Step 3:

[0114] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's device if an anomaly is detected. Input data includes information about the type and location of the anomaly. Output is the notification message displayed on the owner's device. Specifically, the notification content is created and immediate delivery is configured.

[0115] Step 4:

[0116] The user checks notification messages received on the owner's device to understand the vehicle's status. Input is notification information sent from the server. Output is map information and camera footage displayed on the device screen. Specifically, the application displays the vehicle's location on a map and starts video streaming as needed.

[0117] Step 5:

[0118] The server shares security information with other users via the communication network. Inputs include detailed data on anomalies and location information. Outputs include warning notifications and distribution of shared information to other users. Specifically, it transmits information to the local communication network and sends similar warnings to other owners' devices.

[0119] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0120] This invention is a vehicle management system designed to improve vehicle security and comfort, and in particular, to provide a more personalized experience by integrating an emotion engine that recognizes user emotions.

[0121] System Configuration

[0122] 1. Terminal: Sensors and cameras mounted on the vehicle not only perform standard security functions, but also acquire data such as the user's facial expressions and tone of voice to provide to the emotion engine. This allows for real-time monitoring of the user's emotional state.

[0123] 2. Emotion Engine: This engine receives input from the device and analyzes the user's emotional state. Using AI algorithms, this engine identifies different emotions such as stress, anger, and joy, and adaptively reflects them in the in-car environment.

[0124] 3. Server: Based on the analysis results of the emotion engine, the server controls various systems in the vehicle. For example, if the server determines that the user is stressed, it will activate the entertainment system and play relaxing music. It will also display appropriate warnings for driving if necessary.

[0125] 4. User: The in-vehicle environment is automatically adjusted to support a comfortable driving experience. Users can also check emotional engine feedback and recommendations through the application.

[0126] Usage example

[0127] For example, if a user experiences stress during a long drive, the device communicates this stress level to the emotion engine. The emotion engine analyzes the data to determine the level of stress and sends this information to the server. The server then plays relaxation music through the vehicle's audio system and adjusts the lighting and air conditioning to optimal settings. This allows the user to relax, creating a safe and comfortable driving environment.

[0128] In this way, a vehicle management system incorporating an emotional engine functions not merely as a security tool, but as an advanced assistance device that also considers driver comfort.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The terminal uses sensors and cameras mounted on the vehicle to capture the user's facial expressions and voice, and this data is then prepared to be sent to the emotion engine.

[0132] Step 2:

[0133] The emotion engine on the server analyzes the emotion data sent from the device. The emotion engine uses AI algorithms to continuously identify the user's emotions, such as stress, joy, and anger.

[0134] Step 3:

[0135] Based on the analysis results from the emotion engine, the server automatically determines how to adjust the in-car environment to suit the user's current emotional state. Specifically, this includes selecting appropriate music, adjusting the lighting, and setting the air conditioning.

[0136] Step 4:

[0137] The server sends instructions to the terminal to adjust the in-car environment as determined by the server. The terminal then plays relaxation music on the audio system, changes the brightness and color of the cabin lights to a calming tone, and sets the air conditioning temperature to a level that the user finds comfortable.

[0138] Step 5:

[0139] The user experiences the altered in-car environment and continues driving in a relaxed state. The user's emotional changes are continuously monitored, and the server and terminal make further adjustments as needed.

[0140] Step 6:

[0141] Users can check their current emotional state and receive feedback from the server via their smartphones. This allows users to directly experience the benefits of intelligent vehicle adjustments based on their emotions.

[0142] In this way, by recognizing the user's emotions in real time and automatically adjusting the environment accordingly, driving comfort and safety are significantly improved.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0145] To comprehensively improve safety and comfort within vehicles, it is necessary to not only detect anomalies but also to understand the user's emotional state in real time and optimize the in-vehicle environment accordingly. However, conventional systems lacked the means to integrate and efficiently perform these tasks, resulting in difficulties in providing quick and appropriate responses.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for analyzing information acquired by a detection device mounted on the vehicle to detect abnormalities and emotional states; means for notifying the owner of the detected abnormalities and emotional states in real time and automatically adjusting the in-vehicle environment; and means for tracking the vehicle's location using a location determination system and providing that information to the owner. This makes it possible for the user to continue driving comfortably while maintaining the safety of the vehicle.

[0148] A "detection device" is a device, such as a sensor or camera, installed in a vehicle to acquire information about the surroundings and the occupants.

[0149] "Information" refers to data acquired by detection devices, which is used to determine the user's emotional state and any abnormalities in the vehicle.

[0150] An "abnormality" refers to any unusual condition or behavior that may interfere with the normal operation of a vehicle.

[0151] "Emotional state" refers to the user's mental state and includes emotions such as stress, anger, and joy.

[0152] A "positioning system" refers to the technology and devices used to measure the position of a vehicle on Earth, and GPS is the most common example.

[0153] "Owner" refers to the person who owns the vehicle or has the legal right to use it, and who is responsible for the management and operation of the vehicle.

[0154] "In-vehicle environment" refers to the physical and psychological environment inside a vehicle, including temperature, lighting, music, and other factors.

[0155] A "networking method" is a system or technology that allows multiple users to connect with each other and share information.

[0156] This invention is a system that uses sensors and cameras mounted on a vehicle to acquire information about the interior and surroundings of the vehicle, and uses that information to improve the safety and comfort of the vehicle.

[0157] The terminal uses multiple sensors and cameras installed in the vehicle to acquire information such as the user's facial expressions, tone of voice, and any vehicle malfunctions. These devices include reliable voice recognition microphones and high-resolution cameras, enabling real-time monitoring of the user's emotional state and the vehicle's operating status.

[0158] The server utilizes an emotion engine to process information sent from the terminal using AI algorithms and analyze the user's emotions. This analysis identifies emotional states such as stress and joy, and also detects anomalies. Based on these analysis results, the server can integrate and control various systems in the vehicle. For example, if the server determines that the user is stressed, it will control the audio system to play relaxation music. It will also automatically adjust lighting and temperature to provide the user with a comfortable environment.

[0159] Users can enjoy the safety and comfort provided through this system while receiving feedback via a smartphone app or in-car display. This allows users to monitor their own emotional state and the vehicle's safety status, enabling them to continue driving safely and comfortably.

[0160] For example, if a user begins to feel stressed during a long drive, the device detects changes in the user's facial expression, and the server makes a judgment and adjusts the vehicle's entertainment system. This series of actions allows the user to relax and maintains a safe driving environment.

[0161] An example of a prompt using a generative AI model is: "Please describe a vehicle management system that analyzes the user's emotional state. In particular, please explain in detail how sensors and AI algorithms are used to improve user comfort."

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The device uses sensors and cameras mounted on the vehicle to collect data on the user's facial expressions and voice tone. As input, the sensors capture the user's facial movements and voice waveforms. This data serves as foundational information for analyzing the user's emotional state. As output, processed data is generated for transmission to the emotion engine. This processing extracts features related to emotions.

[0165] Step 2:

[0166] The server receives emotional data transmitted from the terminal and analyzes it using an AI algorithm. Feature data is provided as input to the emotion engine. For data processing, a machine learning model is used to classify the user's emotions into categories such as stress, anger, and joy. Tagged data indicating the user's emotional state is generated as output. This information forms the basis for adjusting the in-car environment.

[0167] Step 3:

[0168] The server controls various vehicle systems based on the analyzed emotional state. Tagged data representing the emotional state is used as input. Based on this, the server adaptively adjusts the vehicle's audio system, lighting, and air conditioning. Specifically, if stress is detected, it plays relaxation music and sets the interior lighting to a softer glow. The output is a tuned in-car environment.

[0169] Step 4:

[0170] Users benefit from these automatically adjusted environments, experiencing a more comfortable and safer driving experience inside the vehicle. Users can view current sentiment analysis results and system recommendations through applications and displays. Input includes visual or auditory feedback of the adjusted environment. Output is a state where users can continue driving with confidence.

[0171] (Application Example 2)

[0172] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0173] Conventional vehicle management systems primarily focus on security features, neglecting user comfort and providing a personalized experience. Furthermore, they lack mechanisms to alleviate stress and anxiety associated with long-distance driving. Understanding the user's emotional state and making appropriate environmental adjustments accordingly is crucial. This invention aims to solve these problems and provide a safer and more comfortable driving experience.

[0174] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0175] In this invention, the server includes means for analyzing data acquired by detectors mounted on the vehicle to detect anomalies, means for analyzing the user's emotional state and automatically adjusting the in-vehicle environment accordingly, and means for providing feedback and advice based on the user's emotional state. As a result, the user is not only always provided with a comfortable in-vehicle environment, but driving stress is reduced, enabling safe and personalized travel.

[0176] A "detector" is a device installed in a vehicle that senses the surrounding environment and the state of the occupants, and has the ability to acquire data.

[0177] "Powertrain" is a general term for devices such as engines and electric motors that control the movement of a vehicle.

[0178] The Global Positioning System is a satellite system that accurately tracks the location of vehicles and provides geographical coordinates.

[0179] "Communication methods" refer to methods that use a network to exchange information with other users and share crime prevention information.

[0180] An "emotion analysis device" is hardware or software that analyzes a user's emotional state and outputs it as data.

[0181] "Feedback" is a means of providing users with information and advice based on their analyzed emotional state.

[0182] To implement this invention, a vehicle is equipped with detectors and cameras, and a system is constructed to dynamically control the environment inside the vehicle using the data acquired by these devices. The terminal collects information such as passengers' facial expressions and voice tone through the detectors and cameras, and analyzes this information with an emotion analysis device. This analysis utilizes generative AI models such as TensorFlow and PyTorch to identify the emotional state of the user.

[0183] Based on these analysis results, the server issues commands to adjust the in-car displays, entertainment systems, lighting, and air conditioning. Specifically, if it determines that the user's emotions are unstable, it will play relaxing music or change the interior lighting to a softer color. It also improves comfort by providing feedback to the user with advice based on their emotional state.

[0184] For example, in detecting stress during long-distance driving, if the terminal detects a stressed facial expression, the server automatically displays a scenic image to help the user relax and sends a message such as, "Take a deep breath and relax." Examples of such prompts include, "Suggest a playlist of music that will help the passenger relax," or "Suggest actions to alleviate the passenger's stress."

[0185] This system goes beyond traditional security-focused vehicle management, enabling the provision of personalized driving experiences that are attentive to the user's emotions.

[0186] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0187] Step 1:

[0188] The terminal uses detectors and cameras mounted on the vehicle to acquire passengers' facial expressions and voice tones in real time. This input data includes subtle facial movements and emotional intensities in voice. The acquired data is then transmitted to an emotion analysis device.

[0189] Step 2:

[0190] The server analyzes the user's emotional state based on the input data via an emotion analysis device. TensorFlow, a generative AI model, is used for the analysis to identify emotions such as stress, joy, and anger. As a result of this data processing, the user's current emotional state is output.

[0191] Step 3:

[0192] The server issues commands to adjust the in-car environment based on the analyzed emotional state. For example, if it determines that the user is stressed, it will issue a command to play relaxation music to the in-car audio system. The in-car lighting and air conditioning will also be automatically adjusted. Outputs at this stage include changes to the lighting color settings and the selection of music playlists.

[0193] Step 4:

[0194] The server displays on-screen instructions to the user, showing feedback and prompts. Messages such as "Take a deep breath and relax" or "Why not refresh yourself by listening to your favorite music?" are generated and displayed on the screen. This allows the user to receive instructions to calm their emotions.

[0195] Step 5:

[0196] The user is more likely to take actions that stabilize their emotions by following feedback from the server. For example, they might choose and listen to suggested relaxation music to relax. In this step, responding to prompts from the system is crucial.

[0197] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0198] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0199] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0200] [Second Embodiment]

[0201] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0202] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0203] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0204] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0205] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0206] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0207] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0208] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0209] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0210] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0211] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0212] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0213] The vehicle security system of the present invention is an AI-based system aimed at preventing vehicle theft and tracking, and functions through the coordinated operation of multiple components. The system consists of sensors and cameras on the vehicle itself, a central server, and the owner's terminal (such as a smartphone).

[0214] System Configuration

[0215] 1. Terminals: Sensors and cameras mounted on the vehicle monitor the surrounding environment in real time, constantly checking for any abnormalities. When these devices detect abnormal events such as doors opening and closing, windows breaking, or movement inside the vehicle, they immediately send the information to the server.

[0216] 2. Server: Upon receiving notification of an anomaly, the server analyzes the data to determine the type and severity of the anomaly. If an anomaly is confirmed, a real-time push notification is sent to the owner's smartphone. The server also uses GPS data to determine the vehicle's current location and sends this location information to the device.

[0217] 3. User: Owners can check the vehicle's status from their smartphones via the notifications they receive. The application allows them to view the vehicle's current location on a map and also provides a function to stream video from cameras installed on the vehicle in real time.

[0218] Usage example

[0219] For example, if a vehicle door is forcibly opened due to unauthorized access at night, the terminal immediately detects the anomaly and sends the information to the server. In response, the server sends a notification to the owner's smartphone based on a predetermined protocol. Furthermore, the server automatically shuts down the engine to prevent the vehicle from moving. Meanwhile, the user can open the application to check the vehicle's current location and camera footage of the surrounding area, and report to the police if necessary.

[0220] By building such a system, vehicle theft prevention and rapid tracking are possible, providing owners with peace of mind and safety. Furthermore, sharing crime prevention information with other users improves the overall level of crime prevention in the community. This system is particularly effective for luxury cars, where a high level of security is required.

[0221] The following describes the processing flow.

[0222] Step 1:

[0223] The terminal keeps the vehicle's sensors and cameras running continuously to monitor things like door opening and closing, movement inside the vehicle, and glass breakage. The data obtained from the sensors is analyzed in real time, and if an abnormality beyond the normal range is detected, it is immediately communicated to the server.

[0224] Step 2:

[0225] The server receives abnormal data sent from the terminal and analyzes its contents. It determines the type and severity of the abnormality and identifies the appropriate action to take. The analysis performed in this step determines whether or not the engine shutdown trigger is pulled.

[0226] Step 3:

[0227] If an anomaly is detected by the server, it will immediately send a notification to the owner's device. The notification will include detailed information about the anomaly, the time it occurred, and even real-time location information, which the owner can view on their smartphone.

[0228] Step 4:

[0229] The server uses location services to continuously determine the vehicle's current location. This location information is kept up-to-date and updated on the user's device as needed.

[0230] Step 5:

[0231] Users can check notifications and verify the vehicle's status and current location through a smartphone application. They can also stream real-time video from the vehicle's cameras to visually confirm any suspicious situations.

[0232] Step 6:

[0233] If the malfunction is not resolved, the device will continue to emit an alarm and maintain engine shutdown. This physically prevents unauthorized engine starting or vehicle movement.

[0234] These steps ensure vehicle safety and enable prompt and appropriate responses to users.

[0235] (Example 1)

[0236] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0237] Existing vehicle security systems have challenges in terms of rapid response after anomaly detection and secure vehicle management. Furthermore, they lack sufficient functions to prevent unauthorized access to vehicles and to provide information to owners, making it difficult to prevent damage resulting from such access. In addition, the lack of situational awareness and effective information sharing with other users results in insufficient overall security measures. There is a need to solve these problems and dramatically improve vehicle safety.

[0238] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0239] In this invention, the server includes means for analyzing data acquired by a sensor device and a camera device to detect anomalies, means for analyzing the data on an analysis platform and evaluating the severity of the anomaly, and means for providing video data to the owner in real time using streaming distribution technology. This enables a rapid response after anomaly detection and allows the owner to check and manage the safety of the vehicle in real time.

[0240] A "sensor device" is a device used to detect physical changes in the surrounding environment and is used to detect abnormal behavior or changes.

[0241] A "photography device" is a device used to acquire visual information and is used to understand the situation around a vehicle.

[0242] An "analysis platform" is a foundational technology for receiving, processing, and analyzing data, and is used to evaluate anomalies and determine their severity.

[0243] "Streaming technology" is a technology that continuously transmits video data in real time over the internet and is used to instantly check the situation in remote locations.

[0244] "Power system" refers to the equipment used to drive a vehicle, and mainly includes engines and motors.

[0245] "Location measurement technology" refers to technology used to determine geographical location, and generally involves using GPS or similar systems to identify the location information of a moving object.

[0246] "Communication means" refers to the media and technologies used to send and receive information, and is used for the purpose of sharing crime prevention information with others through a communication network.

[0247] "Personal identification technology" refers to technology used to distinguish a specific person from others, and is used to prevent unauthorized access using biometric information or passwords.

[0248] This system utilizes complex technologies to enhance vehicle safety, handling everything from anomaly detection to information sharing in a consistent manner. Specifically, it is implemented through the following combination of hardware and software.

[0249] The terminal includes a sensor device and a camera device mounted on the vehicle. The sensor device constantly monitors for physical anomalies, and when an anomaly is detected, the camera device acquires the relevant video footage and transmits the data to the server. For example, if a door is opened illegally, the terminal immediately transmits that information to the server.

[0250] The server operates on a cloud infrastructure such as AWS and provides an analysis platform for analyzing received data. This platform utilizes machine learning models such as TensorFlow to determine the severity of anomalies. When an anomaly is identified, Firebase Cloud Messaging is used to immediately notify the owner and, if necessary, to instruct the shutdown of the power unit. In addition, location measurement technology is used to obtain and track the vehicle's current location, and this information is communicated to the user in real time.

[0251] Users can receive notifications via their smartphones and check the vehicle's status and location using a dedicated application. This app integrates with the Google Maps API to display the vehicle's location on a map and provides real-time video using streaming technology. If necessary, this information can be used to immediately report the incident to the police.

[0252] As a concrete example, if unauthorized access to a vehicle occurs at night, the terminal detects the anomaly and sends data to the server. The server analyzes this data and sends a push notification to the user's smartphone, while also displaying the vehicle's current location on a map. The user can then view the camera footage through the application and make a safe assessment of the situation.

[0253] An example of a prompt would be: "Please describe the specific processes involved in an AI-based vehicle security system, from anomaly detection and analysis to sending real-time notifications to the user. Focus on how TensorFlow and Firebase Cloud Messaging are utilized."

[0254] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0255] Step 1:

[0256] The terminal uses sensors and cameras mounted on the vehicle to monitor the surrounding environment in real time. Inputs include capturing the opening and closing status of the vehicle doors, the state of the windows, and movement inside the vehicle. Based on the data from these sensors, it detects anomalies, and if an anomaly is determined, it immediately transmits the sensor data and video from the camera to a central server. The output at this stage specifically consists of the anomaly determination result and video data.

[0257] Step 2:

[0258] The server receives abnormal data and video data transmitted from the terminal. This data, obtained as input, is analyzed using an analysis platform. Specifically, generative AI models and machine learning frameworks are used to classify the data and determine the type and severity of the anomaly. If an anomaly is confirmed as a result of the analysis, the results are formatted as information to notify the owner, and push notification data is generated as output.

[0259] Step 3:

[0260] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's smartphone. This notification includes detailed information about the detected anomaly and the vehicle's current location. The server utilizes the Google Maps API based on the input data to calculate the vehicle's location and add it to the information sent to the owner. The output is a notification message that the owner can immediately recognize.

[0261] Step 4:

[0262] The user checks the push notification received on their smartphone. By launching the application, they can view the vehicle's location on a map in real time and watch the camera footage via streaming. Based on the information received, the user can take prompt action, such as reporting to the police, if necessary. At this stage, the input is notification data from the server, and the output is the specific action taken by the user.

[0263] Step 5:

[0264] If the server determines that an anomaly is serious, it sends a command to the vehicle to shut down the powertrain. The input is the type and severity of the anomaly determined in the previous step, and the output is a control signal that shuts down the engine. This system is designed to minimize the impact of unauthorized access.

[0265] (Application Example 1)

[0266] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] Automobiles are an important means of transportation in daily life, but their theft and misuse are serious problems. Furthermore, systems capable of quickly detecting abnormal vibrations and shocks that occur while a vehicle is in motion and notifying the owner are still insufficient. There is also a need for a system that efficiently shares vehicle security information with other users to improve overall community safety.

[0268] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0269] In this invention, the server includes means for analyzing data acquired by detectors and image sensors mounted on the vehicle to detect anomalies, means for notifying the owner's terminal of the detected anomaly in real time, and means for shutting down the vehicle's power unit based on the detection of an anomaly. This enables vehicle theft prevention, early detection and appropriate response to abnormal events, and rapid sharing of security information.

[0270] A "detector" is a general term for a sensor that is installed on a vehicle and can detect physical changes or abnormalities in the environment.

[0271] An "image sensor" is a device that converts light into electrical signals and acquires surrounding visual information as digital data.

[0272] An "abnormality" refers to a condition that deviates from the normal operation or environmental conditions of a vehicle, including, for example, unauthorized access, unexpected vibrations, or shocks.

[0273] "Owner's terminal" refers to a terminal device such as a mobile phone or tablet used by the vehicle owner to receive various information.

[0274] "Real-time notification" refers to a technological means that allows the owner to be notified almost immediately as soon as an anomaly is detected.

[0275] "Stopping the power system" means forcibly stopping the vehicle's engine and other drive mechanisms to prevent the vehicle from moving illegally.

[0276] A "location information system" is a system that uses the Global Positioning System (GPS) and other technologies to determine and track the current location of a vehicle.

[0277] A "communication network means" is a system that uses network infrastructure for sending and receiving data to share crime prevention information with others.

[0278] "Past operation data" refers to recorded information regarding the vehicle's past driving history and usage.

[0279] "Information to support safe driving" refers to information provided to vehicle owners, including advice and warnings to improve driving behavior and enhance safety.

[0280] Embodiments of this invention consist of a system comprising hardware mounted on a vehicle, a terminal used by the owner, and a central server. This system acquires and analyzes data from detectors and image sensors mounted on the vehicle in real time, thereby enabling efficient security monitoring.

[0281] The server receives the data transmitted from the vehicle and uses TensorFlow, which is software for AI analysis, to detect anomalies. At this time, in order to immediately determine the type and severity of the anomaly and send a notification to the owner's terminal, Firebase Cloud Messaging is utilized. This notification is sent to the owner's terminal in push format, providing a mechanism to respond immediately to anomalies.

[0282] For example, when a detector senses abnormal vibrations that occur when a vehicle collides with another object, the server analyzes the impact data and immediately sends a notification to the owner's terminal saying, "An impact has been detected on the vehicle. Please check the location and video." Also, it is possible to track the current location of the vehicle using a location information system and provide detailed location information to the owner.

[0283] Furthermore, by utilizing communication network means, this security information can also be shared with other users, thus making it possible to improve the security of the entire region. This can be a means of enhancing security awareness, for example, by also notifying nearby users that "Suspicious activity has been detected."

[0284] As a specific application example of this system, there is a prompt sentence for a generative AI model such as "Please tell me the types of sensors and cameras to be added for anomaly detection, and the algorithm by which the AI system can identify anomalies quickly and efficiently." Such prompts have the potential to propose various additional technologies and optimization directions in actual implementation.

[0285] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0286] Step 1:

[0287] The terminal collects various data in real time from detectors and image sensors mounted on the vehicle. Inputs include vibration data and video data, and outputs generate packets for sending this data to a central server. Specifically, it continuously monitors the surrounding environment and, if an anomaly occurs, immediately prepares to assemble it into data and send it.

[0288] Step 2:

[0289] The server receives data sent from the terminal and analyzes its contents. Input data includes vibration intensity and image frames. The server uses TensorFlow to perform analysis to determine whether there are any anomalies in this data. The output generates analysis results regarding the presence and type of anomalies. Specifically, the process involves applying an AI model to the data and calculating a confidence score for the anomalies.

[0290] Step 3:

[0291] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's device if an anomaly is detected. Input data includes information about the type and location of the anomaly. Output is the notification message displayed on the owner's device. Specifically, the notification content is created and immediate delivery is configured.

[0292] Step 4:

[0293] The user checks notification messages received on the owner's device to understand the vehicle's status. Input is notification information sent from the server. Output is map information and camera footage displayed on the device screen. Specifically, the application displays the vehicle's location on a map and starts video streaming as needed.

[0294] Step 5:

[0295] The server shares security information with other users via the communication network. Inputs include detailed data on anomalies and location information. Outputs include warning notifications and distribution of shared information to other users. Specifically, it transmits information to the local communication network and sends similar warnings to other owners' devices.

[0296] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0297] This invention is a vehicle management system designed to improve vehicle security and comfort, and in particular, to provide a more personalized experience by integrating an emotion engine that recognizes user emotions.

[0298] System Configuration

[0299] 1. Terminal: Sensors and cameras mounted on the vehicle not only perform standard security functions, but also acquire data such as the user's facial expressions and tone of voice to provide to the emotion engine. This allows for real-time monitoring of the user's emotional state.

[0300] 2. Emotion Engine: This engine receives input from the device and analyzes the user's emotional state. Using AI algorithms, this engine identifies different emotions such as stress, anger, and joy, and adaptively reflects them in the in-car environment.

[0301] 3. Server: Based on the analysis results of the emotion engine, the server controls various systems in the vehicle. For example, if the server determines that the user is stressed, it will activate the entertainment system and play relaxing music. It will also display appropriate warnings for driving if necessary.

[0302] 4. User: The comfortable driving is supported by automatically adjusting the environment inside the vehicle. The user can also check the feedback and recommendations of the emotion engine through the application.

[0303] Usage example

[0304] For example, when the user feels stressed during a long - time drive, the terminal transmits this stress state to the emotion engine. The emotion engine determines the stress from the given data and sends this information to the server. The server, upon receiving the determination, makes the vehicle's audio system play relaxation music and adjusts the lighting and air - conditioner to optimal settings. As a result, the user relaxes and a safe and comfortable driving environment is realized.

[0305] In this way, the vehicle management system incorporating the emotion engine functions as an advanced support device that takes into account the driver's comfort, rather than just a security tool.

[0306] The following explains the processing flow.

[0307] Step 1:

[0308] The terminal uses sensors and cameras installed in the vehicle to acquire the user's expression and voice, and captures it as emotion data. This data is prepared to be sent to the emotion engine.

[0309] Step 2:

[0310] The emotion engine in the server analyzes the emotion data sent from the terminal. The emotion engine continues to identify emotions such as the user's stress, joy, anger, etc. using AI algorithms.

[0311] [[ID=三十六]]Step 3:

[0312] Based on the analysis results from the emotion engine, the server automatically determines how to adjust the in-car environment to suit the user's current emotional state. Specifically, this includes selecting appropriate music, adjusting the lighting, and setting the air conditioning.

[0313] Step 4:

[0314] The server sends instructions to the terminal to adjust the in-car environment as determined by the server. The terminal then plays relaxation music on the audio system, changes the brightness and color of the cabin lights to a calming tone, and sets the air conditioning temperature to a level that the user finds comfortable.

[0315] Step 5:

[0316] The user experiences the altered in-car environment and continues driving in a relaxed state. The user's emotional changes are continuously monitored, and the server and terminal make further adjustments as needed.

[0317] Step 6:

[0318] Users can check their current emotional state and receive feedback from the server via their smartphones. This allows users to directly experience the benefits of intelligent vehicle adjustments based on their emotions.

[0319] In this way, by recognizing the user's emotions in real time and automatically adjusting the environment accordingly, driving comfort and safety are significantly improved.

[0320] (Example 2)

[0321] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0322] To comprehensively improve safety and comfort within vehicles, it is necessary to not only detect anomalies but also to understand the user's emotional state in real time and optimize the in-vehicle environment accordingly. However, conventional systems lacked the means to integrate and efficiently perform these tasks, resulting in difficulties in providing quick and appropriate responses.

[0323] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0324] In this invention, the server includes means for analyzing information acquired by a detection device mounted on the vehicle to detect abnormalities and emotional states; means for notifying the owner of the detected abnormalities and emotional states in real time and automatically adjusting the in-vehicle environment; and means for tracking the vehicle's location using a location determination system and providing that information to the owner. This makes it possible for the user to continue driving comfortably while maintaining the safety of the vehicle.

[0325] A "detection device" is a device, such as a sensor or camera, installed in a vehicle to acquire information about the surroundings and the occupants.

[0326] "Information" refers to data acquired by detection devices, which is used to determine the user's emotional state and any abnormalities in the vehicle.

[0327] An "abnormality" refers to any unusual condition or behavior that may interfere with the normal operation of a vehicle.

[0328] "Emotional state" refers to the user's mental state and includes emotions such as stress, anger, and joy.

[0329] A "positioning system" refers to the technology and devices used to measure the position of a vehicle on Earth, and GPS is the most common example.

[0330] "Owner" refers to the person who owns the vehicle or has the legal right to use it, and who is responsible for the management and operation of the vehicle.

[0331] "In-vehicle environment" refers to the physical and psychological environment inside a vehicle, including temperature, lighting, music, and other factors.

[0332] A "networking method" is a system or technology that allows multiple users to connect with each other and share information.

[0333] This invention is a system that uses sensors and cameras mounted on a vehicle to acquire information about the interior and surroundings of the vehicle, and uses that information to improve the safety and comfort of the vehicle.

[0334] The terminal uses multiple sensors and cameras installed in the vehicle to acquire information such as the user's facial expressions, tone of voice, and any vehicle malfunctions. These devices include reliable voice recognition microphones and high-resolution cameras, enabling real-time monitoring of the user's emotional state and the vehicle's operating status.

[0335] The server utilizes an emotion engine to process information sent from the terminal using AI algorithms and analyze the user's emotions. This analysis identifies emotional states such as stress and joy, and also detects anomalies. Based on these analysis results, the server can integrate and control various systems in the vehicle. For example, if the server determines that the user is stressed, it will control the audio system to play relaxation music. It will also automatically adjust lighting and temperature to provide the user with a comfortable environment.

[0336] Users can enjoy the safety and comfort provided through this system while receiving feedback via a smartphone app or in-car display. This allows users to monitor their own emotional state and the vehicle's safety status, enabling them to continue driving safely and comfortably.

[0337] For example, if a user begins to feel stressed during a long drive, the device detects changes in the user's facial expression, and the server makes a judgment and adjusts the vehicle's entertainment system. This series of actions allows the user to relax and maintains a safe driving environment.

[0338] An example of a prompt using a generative AI model is: "Please describe a vehicle management system that analyzes the user's emotional state. In particular, please explain in detail how sensors and AI algorithms are used to improve user comfort."

[0339] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0340] Step 1:

[0341] The device uses sensors and cameras mounted on the vehicle to collect data on the user's facial expressions and voice tone. As input, the sensors capture the user's facial movements and voice waveforms. This data serves as foundational information for analyzing the user's emotional state. As output, processed data is generated for transmission to the emotion engine. This processing extracts features related to emotions.

[0342] Step 2:

[0343] The server receives emotional data transmitted from the terminal and analyzes it using an AI algorithm. Feature data is provided as input to the emotion engine. For data processing, a machine learning model is used to classify the user's emotions into categories such as stress, anger, and joy. Tagged data indicating the user's emotional state is generated as output. This information forms the basis for adjusting the in-car environment.

[0344] Step 3:

[0345] The server controls various vehicle systems based on the analyzed emotional state. Tagged data representing the emotional state is used as input. Based on this, the server adaptively adjusts the vehicle's audio system, lighting, and air conditioning. Specifically, if stress is detected, it plays relaxation music and sets the interior lighting to a softer glow. The output is a tuned in-car environment.

[0346] Step 4:

[0347] Users benefit from these automatically adjusted environments, experiencing a more comfortable and safer driving experience inside the vehicle. Users can view current sentiment analysis results and system recommendations through applications and displays. Input includes visual or auditory feedback of the adjusted environment. Output is a state where users can continue driving with confidence.

[0348] (Application Example 2)

[0349] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0350] Conventional vehicle management systems primarily focus on security features, neglecting user comfort and providing a personalized experience. Furthermore, they lack mechanisms to alleviate stress and anxiety associated with long-distance driving. Understanding the user's emotional state and making appropriate environmental adjustments accordingly is crucial. This invention aims to solve these problems and provide a safer and more comfortable driving experience.

[0351] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0352] In this invention, the server includes means for analyzing data acquired by detectors mounted on the vehicle to detect anomalies, means for analyzing the user's emotional state and automatically adjusting the in-vehicle environment accordingly, and means for providing feedback and advice based on the user's emotional state. As a result, the user is not only always provided with a comfortable in-vehicle environment, but driving stress is reduced, enabling safe and personalized travel.

[0353] A "detector" is a device installed in a vehicle that senses the surrounding environment and the state of the occupants, and has the ability to acquire data.

[0354] "Powertrain" is a general term for devices such as engines and electric motors that control the movement of a vehicle.

[0355] The Global Positioning System is a satellite system that accurately tracks the location of vehicles and provides geographical coordinates.

[0356] "Communication methods" refer to methods that use a network to exchange information with other users and share crime prevention information.

[0357] An "emotion analysis device" is hardware or software that analyzes a user's emotional state and outputs it as data.

[0358] "Feedback" is a means of providing users with information and advice based on their analyzed emotional state.

[0359] To implement this invention, a vehicle is equipped with detectors and cameras, and a system is constructed to dynamically control the environment inside the vehicle using the data acquired by these devices. The terminal collects information such as passengers' facial expressions and voice tone through the detectors and cameras, and analyzes this information with an emotion analysis device. This analysis utilizes generative AI models such as TensorFlow and PyTorch to identify the emotional state of the user.

[0360] Based on these analysis results, the server issues commands to adjust the in-car displays, entertainment systems, lighting, and air conditioning. Specifically, if it determines that the user's emotions are unstable, it will play relaxing music or change the interior lighting to a softer color. It also improves comfort by providing feedback to the user with advice based on their emotional state.

[0361] For example, in detecting stress during long-distance driving, if the terminal detects a stressed facial expression, the server automatically displays a scenic image to help the user relax and sends a message such as, "Take a deep breath and relax." Examples of such prompts include, "Suggest a playlist of music that will help the passenger relax," or "Suggest actions to alleviate the passenger's stress."

[0362] This system goes beyond traditional security-focused vehicle management, enabling the provision of personalized driving experiences that are attentive to the user's emotions.

[0363] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0364] Step 1:

[0365] The terminal uses detectors and cameras mounted on the vehicle to acquire passengers' facial expressions and voice tones in real time. This input data includes subtle facial movements and emotional intensities in voice. The acquired data is then transmitted to an emotion analysis device.

[0366] Step 2:

[0367] The server analyzes the user's emotional state based on the input data via an emotion analysis device. TensorFlow, a generative AI model, is used for the analysis to identify emotions such as stress, joy, and anger. As a result of this data processing, the user's current emotional state is output.

[0368] Step 3:

[0369] The server issues commands to adjust the in-car environment based on the analyzed emotional state. For example, if it determines that the user is stressed, it will issue a command to play relaxation music to the in-car audio system. The in-car lighting and air conditioning will also be automatically adjusted. Outputs at this stage include changes to the lighting color settings and the selection of music playlists.

[0370] Step 4:

[0371] The server displays on-screen instructions to the user, showing feedback and prompts. Messages such as "Take a deep breath and relax" or "Why not refresh yourself by listening to your favorite music?" are generated and displayed on the screen. This allows the user to receive instructions to calm their emotions.

[0372] Step 5:

[0373] The user is more likely to take actions that stabilize their emotions by following feedback from the server. For example, they might choose and listen to suggested relaxation music to relax. In this step, responding to prompts from the system is crucial.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0375] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0377] [Third Embodiment]

[0378] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0379] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0380] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0382] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0384] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0385] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0386] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0389] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0390] The vehicle security system of the present invention is an AI-based system aimed at preventing vehicle theft and tracking, and functions through the coordinated operation of multiple components. The system consists of sensors and cameras on the vehicle itself, a central server, and the owner's terminal (such as a smartphone).

[0391] System Configuration

[0392] 1. Terminals: Sensors and cameras mounted on the vehicle monitor the surrounding environment in real time, constantly checking for any abnormalities. When these devices detect abnormal events such as doors opening and closing, windows breaking, or movement inside the vehicle, they immediately send the information to the server.

[0393] 2. Server: Upon receiving notification of an anomaly, the server analyzes the data to determine the type and severity of the anomaly. If an anomaly is confirmed, a real-time push notification is sent to the owner's smartphone. The server also uses GPS data to determine the vehicle's current location and sends this location information to the device.

[0394] 3. User: Owners can check the vehicle's status from their smartphones via the notifications they receive. The application allows them to view the vehicle's current location on a map and also provides a function to stream video from cameras installed on the vehicle in real time.

[0395] Usage example

[0396] For example, if a vehicle door is forcibly opened due to unauthorized access at night, the terminal immediately detects the anomaly and sends the information to the server. In response, the server sends a notification to the owner's smartphone based on a predetermined protocol. Furthermore, the server automatically shuts down the engine to prevent the vehicle from moving. Meanwhile, the user can open the application to check the vehicle's current location and camera footage of the surrounding area, and report to the police if necessary.

[0397] By building such a system, vehicle theft prevention and rapid tracking are possible, providing owners with peace of mind and safety. Furthermore, sharing crime prevention information with other users improves the overall level of crime prevention in the community. This system is particularly effective for luxury cars, where a high level of security is required.

[0398] The following describes the processing flow.

[0399] Step 1:

[0400] The terminal keeps the vehicle's sensors and cameras running continuously to monitor things like door opening and closing, movement inside the vehicle, and glass breakage. The data obtained from the sensors is analyzed in real time, and if an abnormality beyond the normal range is detected, it is immediately communicated to the server.

[0401] Step 2:

[0402] The server receives abnormal data sent from the terminal and analyzes its contents. It determines the type and severity of the abnormality and identifies the appropriate action to take. The analysis performed in this step determines whether or not the engine shutdown trigger is pulled.

[0403] Step 3:

[0404] If an anomaly is detected by the server, it will immediately send a notification to the owner's device. The notification will include detailed information about the anomaly, the time it occurred, and even real-time location information, which the owner can view on their smartphone.

[0405] Step 4:

[0406] The server uses location services to continuously determine the vehicle's current location. This location information is kept up-to-date and updated on the user's device as needed.

[0407] Step 5:

[0408] Users can check notifications and verify the vehicle's status and current location through a smartphone application. They can also stream real-time video from the vehicle's cameras to visually confirm any suspicious situations.

[0409] Step 6:

[0410] If the malfunction is not resolved, the device will continue to emit an alarm and maintain engine shutdown. This physically prevents unauthorized engine starting or vehicle movement.

[0411] These steps ensure vehicle safety and enable prompt and appropriate responses to users.

[0412] (Example 1)

[0413] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0414] Existing vehicle security systems have challenges in terms of rapid response after anomaly detection and secure vehicle management. Furthermore, they lack sufficient functions to prevent unauthorized access to vehicles and to provide information to owners, making it difficult to prevent damage resulting from such access. In addition, the lack of situational awareness and effective information sharing with other users results in insufficient overall security measures. There is a need to solve these problems and dramatically improve vehicle safety.

[0415] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0416] In this invention, the server includes means for analyzing data acquired by a sensor device and a camera device to detect anomalies, means for analyzing the data on an analysis platform and evaluating the severity of the anomaly, and means for providing video data to the owner in real time using streaming distribution technology. This enables a rapid response after anomaly detection and allows the owner to check and manage the safety of the vehicle in real time.

[0417] A "sensor device" is a device used to detect physical changes in the surrounding environment and is used to detect abnormal behavior or changes.

[0418] A "photography device" is a device used to acquire visual information and is used to understand the situation around a vehicle.

[0419] An "analysis platform" is a foundational technology for receiving, processing, and analyzing data, and is used to evaluate anomalies and determine their severity.

[0420] "Streaming technology" is a technology that continuously transmits video data in real time over the internet and is used to instantly check the situation in remote locations.

[0421] "Power system" refers to the equipment used to drive a vehicle, and mainly includes engines and motors.

[0422] "Location measurement technology" refers to technology used to determine geographical location, and generally involves using GPS or similar systems to identify the location information of a moving object.

[0423] "Communication means" refers to the media and technologies used to send and receive information, and is used for the purpose of sharing crime prevention information with others through a communication network.

[0424] "Personal identification technology" refers to technology used to distinguish a specific person from others, and is used to prevent unauthorized access using biometric information or passwords.

[0425] This system utilizes complex technologies to enhance vehicle safety, handling everything from anomaly detection to information sharing in a consistent manner. Specifically, it is implemented through the following combination of hardware and software.

[0426] The terminal includes a sensor device and a camera device mounted on the vehicle. The sensor device constantly monitors for physical anomalies, and when an anomaly is detected, the camera device acquires the relevant video footage and transmits the data to the server. For example, if a door is opened illegally, the terminal immediately transmits that information to the server.

[0427] The server operates on a cloud infrastructure such as AWS and provides an analysis platform for analyzing received data. This platform utilizes machine learning models such as TensorFlow to determine the severity of anomalies. When an anomaly is identified, Firebase Cloud Messaging is used to immediately notify the owner and, if necessary, to instruct the shutdown of the power unit. In addition, location measurement technology is used to obtain and track the vehicle's current location, and this information is communicated to the user in real time.

[0428] Users can receive notifications via their smartphones and check the vehicle's status and location using a dedicated application. This app integrates with the Google Maps API to display the vehicle's location on a map and provides real-time video using streaming technology. If necessary, this information can be used to immediately report the incident to the police.

[0429] As a concrete example, if unauthorized access to a vehicle occurs at night, the terminal detects the anomaly and sends data to the server. The server analyzes this data and sends a push notification to the user's smartphone, while also displaying the vehicle's current location on a map. The user can then view the camera footage through the application and make a safe assessment of the situation.

[0430] An example of a prompt would be: "Please describe the specific processes involved in an AI-based vehicle security system, from anomaly detection and analysis to sending real-time notifications to the user. Focus on how TensorFlow and Firebase Cloud Messaging are utilized."

[0431] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0432] Step 1:

[0433] The terminal uses sensors and cameras mounted on the vehicle to monitor the surrounding environment in real time. Inputs include capturing the opening and closing status of the vehicle doors, the state of the windows, and movement inside the vehicle. Based on the data from these sensors, it detects anomalies, and if an anomaly is determined, it immediately transmits the sensor data and video from the camera to a central server. The output at this stage specifically consists of the anomaly determination result and video data.

[0434] Step 2:

[0435] The server receives abnormal data and video data transmitted from the terminal. This data, obtained as input, is analyzed using an analysis platform. Specifically, generative AI models and machine learning frameworks are used to classify the data and determine the type and severity of the anomaly. If an anomaly is confirmed as a result of the analysis, the results are formatted as information to notify the owner, and push notification data is generated as output.

[0436] Step 3:

[0437] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's smartphone. This notification includes detailed information about the detected anomaly and the vehicle's current location. The server utilizes the Google Maps API based on the input data to calculate the vehicle's location and add it to the information sent to the owner. The output is a notification message that the owner can immediately recognize.

[0438] Step 4:

[0439] The user checks the push notification received on their smartphone. By launching the application, they can view the vehicle's location on a map in real time and watch the camera footage via streaming. Based on the information received, the user can take prompt action, such as reporting to the police, if necessary. At this stage, the input is notification data from the server, and the output is the specific action taken by the user.

[0440] Step 5:

[0441] If the server determines that an anomaly is serious, it sends a command to the vehicle to shut down the powertrain. The input is the type and severity of the anomaly determined in the previous step, and the output is a control signal that shuts down the engine. This system is designed to minimize the impact of unauthorized access.

[0442] (Application Example 1)

[0443] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0444] Automobiles are an important means of transportation in daily life, but their theft and misuse are serious problems. Furthermore, systems capable of quickly detecting abnormal vibrations and shocks that occur while a vehicle is in motion and notifying the owner are still insufficient. There is also a need for a system that efficiently shares vehicle security information with other users to improve overall community safety.

[0445] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0446] In this invention, the server includes means for analyzing data acquired by detectors and image sensors mounted on the vehicle to detect anomalies, means for notifying the owner's terminal of the detected anomaly in real time, and means for shutting down the vehicle's power unit based on the detection of an anomaly. This enables vehicle theft prevention, early detection and appropriate response to abnormal events, and rapid sharing of security information.

[0447] A "detector" is a general term for a sensor that is installed on a vehicle and can detect physical changes or abnormalities in the environment.

[0448] An "image sensor" is a device that converts light into electrical signals and acquires surrounding visual information as digital data.

[0449] An "abnormality" refers to a condition that deviates from the normal operation or environmental conditions of a vehicle, including, for example, unauthorized access, unexpected vibrations, or shocks.

[0450] "Owner's terminal" refers to a terminal device such as a mobile phone or tablet used by the vehicle owner to receive various information.

[0451] "Real-time notification" refers to a technological means that allows the owner to be notified almost immediately as soon as an anomaly is detected.

[0452] "Stopping the power system" means forcibly stopping the vehicle's engine and other drive mechanisms to prevent the vehicle from moving illegally.

[0453] A "location information system" is a system that uses the Global Positioning System (GPS) and other technologies to determine and track the current location of a vehicle.

[0454] A "communication network means" is a system that uses network infrastructure for sending and receiving data to share crime prevention information with others.

[0455] "Past operation data" refers to recorded information regarding the vehicle's past driving history and usage.

[0456] "Information to support safe driving" refers to information provided to vehicle owners, including advice and warnings to improve driving behavior and enhance safety.

[0457] Embodiments of this invention consist of a system comprising hardware mounted on a vehicle, a terminal used by the owner, and a central server. This system acquires and analyzes data from detectors and image sensors mounted on the vehicle in real time, thereby enabling efficient security monitoring.

[0458] The server receives data transmitted from the vehicle and uses TensorFlow, AI analysis software, to detect anomalies. Firebase Cloud Messaging is used to immediately determine the type and severity of the anomaly and send a notification to the owner's device. This notification is sent via push notification to the owner's device, providing a system for immediate response to anomalies.

[0459] For example, if a detector senses abnormal vibrations caused by a collision between a vehicle and another object, the server analyzes the impact data and immediately sends a notification to the owner's device stating, "An impact has been detected on your vehicle. Please check the location and video." Furthermore, it is possible to track the vehicle's current location using a location information system and provide the owner with detailed location information.

[0460] Furthermore, by utilizing communication networks, this crime prevention information can be shared with other users, thereby improving security throughout the entire area. For example, this can raise crime prevention awareness among nearby users by notifying them that "suspicious activity has been detected."

[0461] A concrete example of this system's application is a prompt to the generative AI model such as, "Please tell me what types of sensors and cameras should be added for anomaly detection, and what algorithm the AI ​​system can use to quickly and efficiently identify anomalies." Such prompts have the potential to suggest various additional technologies and optimization directions in actual implementation.

[0462] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0463] Step 1:

[0464] The terminal collects various data in real time from detectors and image sensors mounted on the vehicle. Inputs include vibration data and video data, and outputs generate packets for sending this data to a central server. Specifically, it continuously monitors the surrounding environment and, if an anomaly occurs, immediately prepares to assemble it into data and send it.

[0465] Step 2:

[0466] The server receives data sent from the terminal and analyzes its contents. Input data includes vibration intensity and image frames. The server uses TensorFlow to perform analysis to determine whether there are any anomalies in this data. The output generates analysis results regarding the presence and type of anomalies. Specifically, the process involves applying an AI model to the data and calculating a confidence score for the anomalies.

[0467] Step 3:

[0468] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's device if an anomaly is detected. Input data includes information about the type and location of the anomaly. Output is the notification message displayed on the owner's device. Specifically, the notification content is created and immediate delivery is configured.

[0469] Step 4:

[0470] The user checks notification messages received on the owner's device to understand the vehicle's status. Input is notification information sent from the server. Output is map information and camera footage displayed on the device screen. Specifically, the application displays the vehicle's location on a map and starts video streaming as needed.

[0471] Step 5:

[0472] The server shares security information with other users via the communication network. Inputs include detailed data on anomalies and location information. Outputs include warning notifications and distribution of shared information to other users. Specifically, it transmits information to the local communication network and sends similar warnings to other owners' devices.

[0473] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0474] This invention is a vehicle management system designed to improve vehicle security and comfort, and in particular, to provide a more personalized experience by integrating an emotion engine that recognizes user emotions.

[0475] System Configuration

[0476] 1. Terminal: Sensors and cameras mounted on the vehicle not only perform standard security functions, but also acquire data such as the user's facial expressions and tone of voice to provide to the emotion engine. This allows for real-time monitoring of the user's emotional state.

[0477] 2. Emotion Engine: This engine receives input from the device and analyzes the user's emotional state. Using AI algorithms, this engine identifies different emotions such as stress, anger, and joy, and adaptively reflects them in the in-car environment.

[0478] 3. Server: Based on the analysis results of the emotion engine, the server controls various systems in the vehicle. For example, if the server determines that the user is stressed, it will activate the entertainment system and play relaxing music. It will also display appropriate warnings for driving if necessary.

[0479] 4. User: The in-vehicle environment is automatically adjusted to support a comfortable driving experience. Users can also check emotional engine feedback and recommendations through the application.

[0480] Usage example

[0481] For example, if a user experiences stress during a long drive, the device communicates this stress level to the emotion engine. The emotion engine analyzes the data to determine the level of stress and sends this information to the server. The server then plays relaxation music through the vehicle's audio system and adjusts the lighting and air conditioning to optimal settings. This allows the user to relax, creating a safe and comfortable driving environment.

[0482] In this way, a vehicle management system incorporating an emotional engine functions not merely as a security tool, but as an advanced assistance device that also considers driver comfort.

[0483] The following describes the processing flow.

[0484] Step 1:

[0485] The terminal uses sensors and cameras mounted on the vehicle to capture the user's facial expressions and voice, and this data is then prepared to be sent to the emotion engine.

[0486] Step 2:

[0487] The emotion engine on the server analyzes the emotion data sent from the device. The emotion engine uses AI algorithms to continuously identify the user's emotions, such as stress, joy, and anger.

[0488] Step 3:

[0489] Based on the analysis results from the emotion engine, the server automatically determines how to adjust the in-car environment to suit the user's current emotional state. Specifically, this includes selecting appropriate music, adjusting the lighting, and setting the air conditioning.

[0490] Step 4:

[0491] The server sends instructions to the terminal to adjust the in-car environment as determined by the server. The terminal then plays relaxation music on the audio system, changes the brightness and color of the cabin lights to a calming tone, and sets the air conditioning temperature to a level that the user finds comfortable.

[0492] Step 5:

[0493] The user experiences the altered in-car environment and continues driving in a relaxed state. The user's emotional changes are continuously monitored, and the server and terminal make further adjustments as needed.

[0494] Step 6:

[0495] Users can check their current emotional state and receive feedback from the server via their smartphones. This allows users to directly experience the benefits of intelligent vehicle adjustments based on their emotions.

[0496] In this way, by recognizing the user's emotions in real time and automatically adjusting the environment accordingly, driving comfort and safety are significantly improved.

[0497] (Example 2)

[0498] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0499] To comprehensively improve safety and comfort within vehicles, it is necessary to not only detect anomalies but also to understand the user's emotional state in real time and optimize the in-vehicle environment accordingly. However, conventional systems lacked the means to integrate and efficiently perform these tasks, resulting in difficulties in providing quick and appropriate responses.

[0500] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0501] In this invention, the server includes means for analyzing information acquired by a detection device mounted on the vehicle to detect abnormalities and emotional states; means for notifying the owner of the detected abnormalities and emotional states in real time and automatically adjusting the in-vehicle environment; and means for tracking the vehicle's location using a location determination system and providing that information to the owner. This makes it possible for the user to continue driving comfortably while maintaining the safety of the vehicle.

[0502] A "detection device" is a device, such as a sensor or camera, installed in a vehicle to acquire information about the surroundings and the occupants.

[0503] "Information" refers to data acquired by detection devices, which is used to determine the user's emotional state and any abnormalities in the vehicle.

[0504] An "abnormality" refers to any unusual condition or behavior that may interfere with the normal operation of a vehicle.

[0505] "Emotional state" refers to the user's mental state and includes emotions such as stress, anger, and joy.

[0506] A "positioning system" refers to the technology and devices used to measure the position of a vehicle on Earth, and GPS is the most common example.

[0507] "Owner" refers to the person who owns the vehicle or has the legal right to use it, and who is responsible for the management and operation of the vehicle.

[0508] "In-vehicle environment" refers to the physical and psychological environment inside a vehicle, including temperature, lighting, music, and other factors.

[0509] A "networking method" is a system or technology that allows multiple users to connect with each other and share information.

[0510] This invention is a system that uses sensors and cameras mounted on a vehicle to acquire information about the interior and surroundings of the vehicle, and uses that information to improve the safety and comfort of the vehicle.

[0511] The terminal uses multiple sensors and cameras installed in the vehicle to acquire information such as the user's facial expressions, tone of voice, and any vehicle malfunctions. These devices include reliable voice recognition microphones and high-resolution cameras, enabling real-time monitoring of the user's emotional state and the vehicle's operating status.

[0512] The server utilizes an emotion engine to process information sent from the terminal using AI algorithms and analyze the user's emotions. This analysis identifies emotional states such as stress and joy, and also detects anomalies. Based on these analysis results, the server can integrate and control various systems in the vehicle. For example, if the server determines that the user is stressed, it will control the audio system to play relaxation music. It will also automatically adjust lighting and temperature to provide the user with a comfortable environment.

[0513] Users can enjoy the safety and comfort provided through this system while receiving feedback via a smartphone app or in-car display. This allows users to monitor their own emotional state and the vehicle's safety status, enabling them to continue driving safely and comfortably.

[0514] For example, if a user begins to feel stressed during a long drive, the device detects changes in the user's facial expression, and the server makes a judgment and adjusts the vehicle's entertainment system. This series of actions allows the user to relax and maintains a safe driving environment.

[0515] An example of a prompt using a generative AI model is: "Please describe a vehicle management system that analyzes the user's emotional state. In particular, please explain in detail how sensors and AI algorithms are used to improve user comfort."

[0516] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0517] Step 1:

[0518] The device uses sensors and cameras mounted on the vehicle to collect data on the user's facial expressions and voice tone. As input, the sensors capture the user's facial movements and voice waveforms. This data serves as foundational information for analyzing the user's emotional state. As output, processed data is generated for transmission to the emotion engine. This processing extracts features related to emotions.

[0519] Step 2:

[0520] The server receives emotional data transmitted from the terminal and analyzes it using an AI algorithm. Feature data is provided as input to the emotion engine. For data processing, a machine learning model is used to classify the user's emotions into categories such as stress, anger, and joy. Tagged data indicating the user's emotional state is generated as output. This information forms the basis for adjusting the in-car environment.

[0521] Step 3:

[0522] The server controls various vehicle systems based on the analyzed emotional state. Tagged data representing the emotional state is used as input. Based on this, the server adaptively adjusts the vehicle's audio system, lighting, and air conditioning. Specifically, if stress is detected, it plays relaxation music and sets the interior lighting to a softer glow. The output is a tuned in-car environment.

[0523] Step 4:

[0524] Users benefit from these automatically adjusted environments, experiencing a more comfortable and safer driving experience inside the vehicle. Users can view current sentiment analysis results and system recommendations through applications and displays. Input includes visual or auditory feedback of the adjusted environment. Output is a state where users can continue driving with confidence.

[0525] (Application Example 2)

[0526] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] Conventional vehicle management systems primarily focus on security features, neglecting user comfort and providing a personalized experience. Furthermore, they lack mechanisms to alleviate stress and anxiety associated with long-distance driving. Understanding the user's emotional state and making appropriate environmental adjustments accordingly is crucial. This invention aims to solve these problems and provide a safer and more comfortable driving experience.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0529] In this invention, the server includes means for analyzing data acquired by detectors mounted on the vehicle to detect anomalies, means for analyzing the user's emotional state and automatically adjusting the in-vehicle environment accordingly, and means for providing feedback and advice based on the user's emotional state. As a result, the user is not only always provided with a comfortable in-vehicle environment, but driving stress is reduced, enabling safe and personalized travel.

[0530] A "detector" is a device installed in a vehicle that senses the surrounding environment and the state of the occupants, and has the ability to acquire data.

[0531] "Powertrain" is a general term for devices such as engines and electric motors that control the movement of a vehicle.

[0532] The Global Positioning System is a satellite system that accurately tracks the location of vehicles and provides geographical coordinates.

[0533] "Communication methods" refer to methods that use a network to exchange information with other users and share crime prevention information.

[0534] An "emotion analysis device" is hardware or software that analyzes a user's emotional state and outputs it as data.

[0535] "Feedback" is a means of providing users with information and advice based on their analyzed emotional state.

[0536] To implement this invention, a vehicle is equipped with detectors and cameras, and a system is constructed to dynamically control the environment inside the vehicle using the data acquired by these devices. The terminal collects information such as passengers' facial expressions and voice tone through the detectors and cameras, and analyzes this information with an emotion analysis device. This analysis utilizes generative AI models such as TensorFlow and PyTorch to identify the emotional state of the user.

[0537] Based on these analysis results, the server issues commands to adjust the in-car displays, entertainment systems, lighting, and air conditioning. Specifically, if it determines that the user's emotions are unstable, it will play relaxing music or change the interior lighting to a softer color. It also improves comfort by providing feedback to the user with advice based on their emotional state.

[0538] For example, in detecting stress during long-distance driving, if the terminal detects a stressed facial expression, the server automatically displays a scenic image to help the user relax and sends a message such as, "Take a deep breath and relax." Examples of such prompts include, "Suggest a playlist of music that will help the passenger relax," or "Suggest actions to alleviate the passenger's stress."

[0539] This system goes beyond traditional security-focused vehicle management, enabling the provision of personalized driving experiences that are attentive to the user's emotions.

[0540] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0541] Step 1:

[0542] The terminal uses detectors and cameras mounted on the vehicle to acquire passengers' facial expressions and voice tones in real time. This input data includes subtle facial movements and emotional intensities in voice. The acquired data is then transmitted to an emotion analysis device.

[0543] Step 2:

[0544] The server analyzes the user's emotional state based on the input data via an emotion analysis device. TensorFlow, a generative AI model, is used for the analysis to identify emotions such as stress, joy, and anger. As a result of this data processing, the user's current emotional state is output.

[0545] Step 3:

[0546] The server issues commands to adjust the in-car environment based on the analyzed emotional state. For example, if it determines that the user is stressed, it will issue a command to play relaxation music to the in-car audio system. The in-car lighting and air conditioning will also be automatically adjusted. Outputs at this stage include changes to the lighting color settings and the selection of music playlists.

[0547] Step 4:

[0548] The server displays on-screen instructions to the user, showing feedback and prompts. Messages such as "Take a deep breath and relax" or "Why not refresh yourself by listening to your favorite music?" are generated and displayed on the screen. This allows the user to receive instructions to calm their emotions.

[0549] Step 5:

[0550] The user is more likely to take actions that stabilize their emotions by following feedback from the server. For example, they might choose and listen to suggested relaxation music to relax. In this step, responding to prompts from the system is crucial.

[0551] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0552] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0553] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0554] [Fourth Embodiment]

[0555] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0556] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0557] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0558] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0559] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0560] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0561] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0562] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0563] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0564] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0567] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0568] The vehicle security system of the present invention is an AI-based system aimed at preventing vehicle theft and tracking, and functions through the coordinated operation of multiple components. The system consists of sensors and cameras on the vehicle itself, a central server, and the owner's terminal (such as a smartphone).

[0569] System Configuration

[0570] 1. Terminals: Sensors and cameras mounted on the vehicle monitor the surrounding environment in real time, constantly checking for any abnormalities. When these devices detect abnormal events such as doors opening and closing, windows breaking, or movement inside the vehicle, they immediately send the information to the server.

[0571] 2. Server: Upon receiving notification of an anomaly, the server analyzes the data to determine the type and severity of the anomaly. If an anomaly is confirmed, a real-time push notification is sent to the owner's smartphone. The server also uses GPS data to determine the vehicle's current location and sends this location information to the device.

[0572] 3. User: Owners can check the vehicle's status from their smartphones via the notifications they receive. The application allows them to view the vehicle's current location on a map and also provides a function to stream video from cameras installed on the vehicle in real time.

[0573] Usage example

[0574] For example, if a vehicle door is forcibly opened due to unauthorized access at night, the terminal immediately detects the anomaly and sends the information to the server. In response, the server sends a notification to the owner's smartphone based on a predetermined protocol. Furthermore, the server automatically shuts down the engine to prevent the vehicle from moving. Meanwhile, the user can open the application to check the vehicle's current location and camera footage of the surrounding area, and report to the police if necessary.

[0575] By building such a system, vehicle theft prevention and rapid tracking are possible, providing owners with peace of mind and safety. Furthermore, sharing crime prevention information with other users improves the overall level of crime prevention in the community. This system is particularly effective for luxury cars, where a high level of security is required.

[0576] The following describes the processing flow.

[0577] Step 1:

[0578] The terminal keeps the vehicle's sensors and cameras running continuously to monitor things like door opening and closing, movement inside the vehicle, and glass breakage. The data obtained from the sensors is analyzed in real time, and if an abnormality beyond the normal range is detected, it is immediately communicated to the server.

[0579] Step 2:

[0580] The server receives abnormal data sent from the terminal and analyzes its contents. It determines the type and severity of the abnormality and identifies the appropriate action to take. The analysis performed in this step determines whether or not the engine shutdown trigger is pulled.

[0581] Step 3:

[0582] If an anomaly is detected by the server, it will immediately send a notification to the owner's device. The notification will include detailed information about the anomaly, the time it occurred, and even real-time location information, which the owner can view on their smartphone.

[0583] Step 4:

[0584] The server uses location services to continuously determine the vehicle's current location. This location information is kept up-to-date and updated on the user's device as needed.

[0585] Step 5:

[0586] Users can check notifications and verify the vehicle's status and current location through a smartphone application. They can also stream real-time video from the vehicle's cameras to visually confirm any suspicious situations.

[0587] Step 6:

[0588] If the malfunction is not resolved, the device will continue to emit an alarm and maintain engine shutdown. This physically prevents unauthorized engine starting or vehicle movement.

[0589] These steps ensure vehicle safety and enable prompt and appropriate responses to users.

[0590] (Example 1)

[0591] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0592] Existing vehicle security systems have challenges in terms of rapid response after anomaly detection and secure vehicle management. Furthermore, they lack sufficient functions to prevent unauthorized access to vehicles and to provide information to owners, making it difficult to prevent damage resulting from such access. In addition, the lack of situational awareness and effective information sharing with other users results in insufficient overall security measures. There is a need to solve these problems and dramatically improve vehicle safety.

[0593] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0594] In this invention, the server includes means for analyzing data acquired by a sensor device and a camera device to detect anomalies, means for analyzing the data on an analysis platform and evaluating the severity of the anomaly, and means for providing video data to the owner in real time using streaming distribution technology. This enables a rapid response after anomaly detection and allows the owner to check and manage the safety of the vehicle in real time.

[0595] A "sensor device" is a device used to detect physical changes in the surrounding environment and is used to detect abnormal behavior or changes.

[0596] A "photography device" is a device used to acquire visual information and is used to understand the situation around a vehicle.

[0597] An "analysis platform" is a foundational technology for receiving, processing, and analyzing data, and is used to evaluate anomalies and determine their severity.

[0598] "Streaming technology" is a technology that continuously transmits video data in real time over the internet and is used to instantly check the situation in remote locations.

[0599] "Power system" refers to the equipment used to drive a vehicle, and mainly includes engines and motors.

[0600] "Location measurement technology" refers to technology used to determine geographical location, and generally involves using GPS or similar systems to identify the location information of a moving object.

[0601] "Communication means" refers to the media and technologies used to send and receive information, and is used for the purpose of sharing crime prevention information with others through a communication network.

[0602] "Personal identification technology" refers to technology used to distinguish a specific person from others, and is used to prevent unauthorized access using biometric information or passwords.

[0603] This system utilizes complex technologies to enhance vehicle safety, handling everything from anomaly detection to information sharing in a consistent manner. Specifically, it is implemented through the following combination of hardware and software.

[0604] The terminal includes a sensor device and a camera device mounted on the vehicle. The sensor device constantly monitors for physical anomalies, and when an anomaly is detected, the camera device acquires the relevant video footage and transmits the data to the server. For example, if a door is opened illegally, the terminal immediately transmits that information to the server.

[0605] The server operates on a cloud infrastructure such as AWS and provides an analysis platform for analyzing received data. This platform utilizes machine learning models such as TensorFlow to determine the severity of anomalies. When an anomaly is identified, Firebase Cloud Messaging is used to immediately notify the owner and, if necessary, to instruct the shutdown of the power unit. In addition, location measurement technology is used to obtain and track the vehicle's current location, and this information is communicated to the user in real time.

[0606] Users can receive notifications via their smartphones and check the vehicle's status and location using a dedicated application. This app integrates with the Google Maps API to display the vehicle's location on a map and provides real-time video using streaming technology. If necessary, this information can be used to immediately report the incident to the police.

[0607] As a concrete example, if unauthorized access to a vehicle occurs at night, the terminal detects the anomaly and sends data to the server. The server analyzes this data and sends a push notification to the user's smartphone, while also displaying the vehicle's current location on a map. The user can then view the camera footage through the application and make a safe assessment of the situation.

[0608] An example of a prompt would be: "Please describe the specific processes involved in an AI-based vehicle security system, from anomaly detection and analysis to sending real-time notifications to the user. Focus on how TensorFlow and Firebase Cloud Messaging are utilized."

[0609] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0610] Step 1:

[0611] The terminal uses sensors and cameras mounted on the vehicle to monitor the surrounding environment in real time. Inputs include capturing the opening and closing status of the vehicle doors, the state of the windows, and movement inside the vehicle. Based on the data from these sensors, it detects anomalies, and if an anomaly is determined, it immediately transmits the sensor data and video from the camera to a central server. The output at this stage specifically consists of the anomaly determination result and video data.

[0612] Step 2:

[0613] The server receives abnormal data and video data transmitted from the terminal. This data, obtained as input, is analyzed using an analysis platform. Specifically, generative AI models and machine learning frameworks are used to classify the data and determine the type and severity of the anomaly. If an anomaly is confirmed as a result of the analysis, the results are formatted as information to notify the owner, and push notification data is generated as output.

[0614] Step 3:

[0615] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's smartphone. This notification includes detailed information about the detected anomaly and the vehicle's current location. The server utilizes the Google Maps API based on the input data to calculate the vehicle's location and add it to the information sent to the owner. The output is a notification message that the owner can immediately recognize.

[0616] Step 4:

[0617] The user checks the push notification received on their smartphone. By launching the application, they can view the vehicle's location on a map in real time and watch the camera footage via streaming. Based on the information received, the user can take prompt action, such as reporting to the police, if necessary. At this stage, the input is notification data from the server, and the output is the specific action taken by the user.

[0618] Step 5:

[0619] If the server determines that an anomaly is serious, it sends a command to the vehicle to shut down the powertrain. The input is the type and severity of the anomaly determined in the previous step, and the output is a control signal that shuts down the engine. This system is designed to minimize the impact of unauthorized access.

[0620] (Application Example 1)

[0621] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] Automobiles are an important means of transportation in daily life, but their theft and misuse are serious problems. Furthermore, systems capable of quickly detecting abnormal vibrations and shocks that occur while a vehicle is in motion and notifying the owner are still insufficient. There is also a need for a system that efficiently shares vehicle security information with other users to improve overall community safety.

[0623] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0624] In this invention, the server includes means for analyzing data acquired by detectors and image sensors mounted on the vehicle to detect anomalies, means for notifying the owner's terminal of the detected anomaly in real time, and means for shutting down the vehicle's power unit based on the detection of an anomaly. This enables vehicle theft prevention, early detection and appropriate response to abnormal events, and rapid sharing of security information.

[0625] A "detector" is a general term for a sensor that is installed on a vehicle and can detect physical changes or abnormalities in the environment.

[0626] An "image sensor" is a device that converts light into electrical signals and acquires surrounding visual information as digital data.

[0627] An "abnormality" refers to a condition that deviates from the normal operation or environmental conditions of a vehicle, including, for example, unauthorized access, unexpected vibrations, or shocks.

[0628] "Owner's terminal" refers to a terminal device such as a mobile phone or tablet used by the vehicle owner to receive various information.

[0629] "Real-time notification" refers to a technological means that allows the owner to be notified almost immediately as soon as an anomaly is detected.

[0630] "Stopping the power system" means forcibly stopping the vehicle's engine and other drive mechanisms to prevent the vehicle from moving illegally.

[0631] A "location information system" is a system that uses the Global Positioning System (GPS) and other technologies to determine and track the current location of a vehicle.

[0632] A "communication network means" is a system that uses network infrastructure for sending and receiving data to share crime prevention information with others.

[0633] "Past operation data" refers to recorded information regarding the vehicle's past driving history and usage.

[0634] "Information to support safe driving" refers to information provided to vehicle owners, including advice and warnings to improve driving behavior and enhance safety.

[0635] Embodiments of this invention consist of a system comprising hardware mounted on a vehicle, a terminal used by the owner, and a central server. This system acquires and analyzes data from detectors and image sensors mounted on the vehicle in real time, thereby enabling efficient security monitoring.

[0636] The server receives data transmitted from the vehicle and uses TensorFlow, AI analysis software, to detect anomalies. Firebase Cloud Messaging is used to immediately determine the type and severity of the anomaly and send a notification to the owner's device. This notification is sent via push notification to the owner's device, providing a system for immediate response to anomalies.

[0637] For example, if a detector senses abnormal vibrations caused by a collision between a vehicle and another object, the server analyzes the impact data and immediately sends a notification to the owner's device stating, "An impact has been detected on your vehicle. Please check the location and video." Furthermore, it is possible to track the vehicle's current location using a location information system and provide the owner with detailed location information.

[0638] Furthermore, by utilizing communication networks, this crime prevention information can be shared with other users, thereby improving security throughout the entire area. For example, this can raise crime prevention awareness among nearby users by notifying them that "suspicious activity has been detected."

[0639] A concrete example of this system's application is a prompt to the generative AI model such as, "Please tell me what types of sensors and cameras should be added for anomaly detection, and what algorithm the AI ​​system can use to quickly and efficiently identify anomalies." Such prompts have the potential to suggest various additional technologies and optimization directions in actual implementation.

[0640] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0641] Step 1:

[0642] The terminal collects various data in real time from detectors and image sensors mounted on the vehicle. Inputs include vibration data and video data, and outputs generate packets for sending this data to a central server. Specifically, it continuously monitors the surrounding environment and, if an anomaly occurs, immediately prepares to assemble it into data and send it.

[0643] Step 2:

[0644] The server receives data sent from the terminal and analyzes its contents. Input data includes vibration intensity and image frames. The server uses TensorFlow to perform analysis to determine whether there are any anomalies in this data. The output generates analysis results regarding the presence and type of anomalies. Specifically, the process involves applying an AI model to the data and calculating a confidence score for the anomalies.

[0645] Step 3:

[0646] Based on the analysis results, the server uses Firebase Cloud Messaging to send a push notification to the owner's device if an anomaly is detected. Input data includes information about the type and location of the anomaly. Output is the notification message displayed on the owner's device. Specifically, the notification content is created and immediate delivery is configured.

[0647] Step 4:

[0648] The user checks notification messages received on the owner's device to understand the vehicle's status. Input is notification information sent from the server. Output is map information and camera footage displayed on the device screen. Specifically, the application displays the vehicle's location on a map and starts video streaming as needed.

[0649] Step 5:

[0650] The server shares security information with other users via the communication network. Inputs include detailed data on anomalies and location information. Outputs include warning notifications and distribution of shared information to other users. Specifically, it transmits information to the local communication network and sends similar warnings to other owners' devices.

[0651] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0652] This invention is a vehicle management system designed to improve vehicle security and comfort, and in particular, to provide a more personalized experience by integrating an emotion engine that recognizes user emotions.

[0653] System Configuration

[0654] 1. Terminal: Sensors and cameras mounted on the vehicle not only perform standard security functions, but also acquire data such as the user's facial expressions and tone of voice to provide to the emotion engine. This allows for real-time monitoring of the user's emotional state.

[0655] 2. Emotion Engine: This engine receives input from the device and analyzes the user's emotional state. Using AI algorithms, this engine identifies different emotions such as stress, anger, and joy, and adaptively reflects them in the in-car environment.

[0656] 3. Server: Based on the analysis results of the emotion engine, the server controls various systems in the vehicle. For example, if the server determines that the user is stressed, it will activate the entertainment system and play relaxing music. It will also display appropriate warnings for driving if necessary.

[0657] 4. User: The in-vehicle environment is automatically adjusted to support a comfortable driving experience. Users can also check emotional engine feedback and recommendations through the application.

[0658] Usage example

[0659] For example, if a user experiences stress during a long drive, the device communicates this stress level to the emotion engine. The emotion engine analyzes the data to determine the level of stress and sends this information to the server. The server then plays relaxation music through the vehicle's audio system and adjusts the lighting and air conditioning to optimal settings. This allows the user to relax, creating a safe and comfortable driving environment.

[0660] In this way, a vehicle management system incorporating an emotional engine functions not merely as a security tool, but as an advanced assistance device that also considers driver comfort.

[0661] The following describes the processing flow.

[0662] Step 1:

[0663] The terminal uses sensors and cameras mounted on the vehicle to capture the user's facial expressions and voice, and this data is then prepared to be sent to the emotion engine.

[0664] Step 2:

[0665] The emotion engine on the server analyzes the emotion data sent from the device. The emotion engine uses AI algorithms to continuously identify the user's emotions, such as stress, joy, and anger.

[0666] Step 3:

[0667] Based on the analysis results from the emotion engine, the server automatically determines how to adjust the in-car environment to suit the user's current emotional state. Specifically, this includes selecting appropriate music, adjusting the lighting, and setting the air conditioning.

[0668] Step 4:

[0669] The server sends instructions to the terminal to adjust the in-car environment as determined by the server. The terminal then plays relaxation music on the audio system, changes the brightness and color of the cabin lights to a calming tone, and sets the air conditioning temperature to a level that the user finds comfortable.

[0670] Step 5:

[0671] The user experiences the altered in-car environment and continues driving in a relaxed state. The user's emotional changes are continuously monitored, and the server and terminal make further adjustments as needed.

[0672] Step 6:

[0673] Users can check their current emotional state and receive feedback from the server via their smartphones. This allows users to directly experience the benefits of intelligent vehicle adjustments based on their emotions.

[0674] In this way, by recognizing the user's emotions in real time and automatically adjusting the environment accordingly, driving comfort and safety are significantly improved.

[0675] (Example 2)

[0676] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0677] To comprehensively improve safety and comfort within vehicles, it is necessary to not only detect anomalies but also to understand the user's emotional state in real time and optimize the in-vehicle environment accordingly. However, conventional systems lacked the means to integrate and efficiently perform these tasks, resulting in difficulties in providing quick and appropriate responses.

[0678] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0679] In this invention, the server includes means for analyzing information acquired by a detection device mounted on the vehicle to detect abnormalities and emotional states; means for notifying the owner of the detected abnormalities and emotional states in real time and automatically adjusting the in-vehicle environment; and means for tracking the vehicle's location using a location determination system and providing that information to the owner. This makes it possible for the user to continue driving comfortably while maintaining the safety of the vehicle.

[0680] A "detection device" is a device, such as a sensor or camera, installed in a vehicle to acquire information about the surroundings and the occupants.

[0681] "Information" refers to data acquired by detection devices, which is used to determine the user's emotional state and any abnormalities in the vehicle.

[0682] An "abnormality" refers to any unusual condition or behavior that may interfere with the normal operation of a vehicle.

[0683] "Emotional state" refers to the user's mental state and includes emotions such as stress, anger, and joy.

[0684] A "positioning system" refers to the technology and devices used to measure the position of a vehicle on Earth, and GPS is the most common example.

[0685] "Owner" refers to the person who owns the vehicle or has the legal right to use it, and who is responsible for the management and operation of the vehicle.

[0686] "In-vehicle environment" refers to the physical and psychological environment inside a vehicle, including temperature, lighting, music, and other factors.

[0687] A "networking method" is a system or technology that allows multiple users to connect with each other and share information.

[0688] This invention is a system that uses sensors and cameras mounted on a vehicle to acquire information about the interior and surroundings of the vehicle, and uses that information to improve the safety and comfort of the vehicle.

[0689] The terminal uses multiple sensors and cameras installed in the vehicle to acquire information such as the user's facial expressions, tone of voice, and any vehicle malfunctions. These devices include reliable voice recognition microphones and high-resolution cameras, enabling real-time monitoring of the user's emotional state and the vehicle's operating status.

[0690] The server utilizes an emotion engine to process information sent from the terminal using AI algorithms and analyze the user's emotions. This analysis identifies emotional states such as stress and joy, and also detects anomalies. Based on these analysis results, the server can integrate and control various systems in the vehicle. For example, if the server determines that the user is stressed, it will control the audio system to play relaxation music. It will also automatically adjust lighting and temperature to provide the user with a comfortable environment.

[0691] Users can enjoy the safety and comfort provided through this system while receiving feedback via a smartphone app or in-car display. This allows users to monitor their own emotional state and the vehicle's safety status, enabling them to continue driving safely and comfortably.

[0692] For example, if a user begins to feel stressed during a long drive, the device detects changes in the user's facial expression, and the server makes a judgment and adjusts the vehicle's entertainment system. This series of actions allows the user to relax and maintains a safe driving environment.

[0693] An example of a prompt using a generative AI model is: "Please describe a vehicle management system that analyzes the user's emotional state. In particular, please explain in detail how sensors and AI algorithms are used to improve user comfort."

[0694] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0695] Step 1:

[0696] The device uses sensors and cameras mounted on the vehicle to collect data on the user's facial expressions and voice tone. As input, the sensors capture the user's facial movements and voice waveforms. This data serves as foundational information for analyzing the user's emotional state. As output, processed data is generated for transmission to the emotion engine. This processing extracts features related to emotions.

[0697] Step 2:

[0698] The server receives emotional data transmitted from the terminal and analyzes it using an AI algorithm. Feature data is provided as input to the emotion engine. For data processing, a machine learning model is used to classify the user's emotions into categories such as stress, anger, and joy. Tagged data indicating the user's emotional state is generated as output. This information forms the basis for adjusting the in-car environment.

[0699] Step 3:

[0700] The server controls various vehicle systems based on the analyzed emotional state. Tagged data representing the emotional state is used as input. Based on this, the server adaptively adjusts the vehicle's audio system, lighting, and air conditioning. Specifically, if stress is detected, it plays relaxation music and sets the interior lighting to a softer glow. The output is a tuned in-car environment.

[0701] Step 4:

[0702] Users benefit from these automatically adjusted environments, experiencing a more comfortable and safer driving experience inside the vehicle. Users can view current sentiment analysis results and system recommendations through applications and displays. Input includes visual or auditory feedback of the adjusted environment. Output is a state where users can continue driving with confidence.

[0703] (Application Example 2)

[0704] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0705] Conventional vehicle management systems primarily focus on security features, neglecting user comfort and providing a personalized experience. Furthermore, they lack mechanisms to alleviate stress and anxiety associated with long-distance driving. Understanding the user's emotional state and making appropriate environmental adjustments accordingly is crucial. This invention aims to solve these problems and provide a safer and more comfortable driving experience.

[0706] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0707] In this invention, the server includes means for analyzing data acquired by detectors mounted on the vehicle to detect anomalies, means for analyzing the user's emotional state and automatically adjusting the in-vehicle environment accordingly, and means for providing feedback and advice based on the user's emotional state. As a result, the user is not only always provided with a comfortable in-vehicle environment, but driving stress is reduced, enabling safe and personalized travel.

[0708] A "detector" is a device installed in a vehicle that senses the surrounding environment and the state of the occupants, and has the ability to acquire data.

[0709] "Powertrain" is a general term for devices such as engines and electric motors that control the movement of a vehicle.

[0710] The Global Positioning System is a satellite system that accurately tracks the location of vehicles and provides geographical coordinates.

[0711] "Communication methods" refer to methods that use a network to exchange information with other users and share crime prevention information.

[0712] An "emotion analysis device" is hardware or software that analyzes a user's emotional state and outputs it as data.

[0713] "Feedback" is a means of providing users with information and advice based on their analyzed emotional state.

[0714] To implement this invention, a vehicle is equipped with detectors and cameras, and a system is constructed to dynamically control the environment inside the vehicle using the data acquired by these devices. The terminal collects information such as passengers' facial expressions and voice tone through the detectors and cameras, and analyzes this information with an emotion analysis device. This analysis utilizes generative AI models such as TensorFlow and PyTorch to identify the emotional state of the user.

[0715] Based on these analysis results, the server issues commands to adjust the in-car displays, entertainment systems, lighting, and air conditioning. Specifically, if it determines that the user's emotions are unstable, it will play relaxing music or change the interior lighting to a softer color. It also improves comfort by providing feedback to the user with advice based on their emotional state.

[0716] For example, in detecting stress during long-distance driving, if the terminal detects a stressed facial expression, the server automatically displays a scenic image to help the user relax and sends a message such as, "Take a deep breath and relax." Examples of such prompts include, "Suggest a playlist of music that will help the passenger relax," or "Suggest actions to alleviate the passenger's stress."

[0717] This system goes beyond traditional security-focused vehicle management, enabling the provision of personalized driving experiences that are attentive to the user's emotions.

[0718] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0719] Step 1:

[0720] The terminal uses detectors and cameras mounted on the vehicle to acquire passengers' facial expressions and voice tones in real time. This input data includes subtle facial movements and emotional intensities in voice. The acquired data is then transmitted to an emotion analysis device.

[0721] Step 2:

[0722] The server analyzes the user's emotional state based on the input data via an emotion analysis device. TensorFlow, a generative AI model, is used for the analysis to identify emotions such as stress, joy, and anger. As a result of this data processing, the user's current emotional state is output.

[0723] Step 3:

[0724] The server issues commands to adjust the in-car environment based on the analyzed emotional state. For example, if it determines that the user is stressed, it will issue a command to play relaxation music to the in-car audio system. The in-car lighting and air conditioning will also be automatically adjusted. Outputs at this stage include changes to the lighting color settings and the selection of music playlists.

[0725] Step 4:

[0726] The server displays on-screen instructions to the user, showing feedback and prompts. Messages such as "Take a deep breath and relax" or "Why not refresh yourself by listening to your favorite music?" are generated and displayed on the screen. This allows the user to receive instructions to calm their emotions.

[0727] Step 5:

[0728] The user is more likely to take actions that stabilize their emotions by following feedback from the server. For example, they might choose and listen to suggested relaxation music to relax. In this step, responding to prompts from the system is crucial.

[0729] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0730] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0731] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0732] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0733] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0734] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0735] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0736] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0737] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0738] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0739] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0740] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0741] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0742] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0743] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0744] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0745] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0746] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0747] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0748] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0749] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0750] The following is further disclosed regarding the embodiments described above.

[0751] (Claim 1)

[0752] A means for detecting anomalies by analyzing data acquired by sensors and cameras mounted on the vehicle,

[0753] A means of notifying the owner of detected anomalies in real time,

[0754] A means for shutting down the vehicle's engine based on the detection of an anomaly,

[0755] A means of tracking the vehicle's location using GPS and providing that information to the owner,

[0756] A networking method for collaboratively implementing crime prevention measures by sharing crime prevention information with other users,

[0757] A means of providing owners with information to support safe driving based on past driving data,

[0758] A system that includes this.

[0759] (Claim 2)

[0760] The system according to claim 1, further comprising means for checking the situation around the vehicle based on data from sensors and cameras and transmitting video to the owner in real time.

[0761] (Claim 3)

[0762] The system according to claim 1, further comprising means for controlling the vehicle's engine to prevent it from being started illegally using biometric authentication.

[0763] "Example 1"

[0764] (Claim 1)

[0765] A means for detecting anomalies by analyzing data acquired by a sensor device and an imaging device,

[0766] A means of notifying the owner of detected anomalies in real time,

[0767] A means for stopping the vehicle's power unit based on the detection of an abnormality,

[0768] A means of tracking the vehicle's location using positioning technology and providing that information to the owner,

[0769] A communication method for collaboratively implementing crime prevention measures by sharing crime prevention information with other users,

[0770] A means of providing owners with information to support safe driving based on past driving data,

[0771] A means of analyzing data on an analysis platform and evaluating the severity of anomalies,

[0772] A means of providing video data to the owner in real time using streaming distribution technology,

[0773] A system that includes this.

[0774] (Claim 2)

[0775] The system according to claim 1, further comprising means for checking the conditions around a vehicle based on data from a sensor device and a camera device, and transmitting video to the owner in real time.

[0776] (Claim 3)

[0777] The system according to claim 1, further comprising means for controlling the vehicle's power unit to prevent it from being started illegally using personal identification technology.

[0778] "Application Example 1"

[0779] (Claim 1)

[0780] A means for detecting anomalies by analyzing data acquired by detectors and image sensors mounted on the vehicle,

[0781] A means of notifying the owner's device in real time of detected anomalies,

[0782] A means for stopping the vehicle's power system based on the detection of an abnormality,

[0783] A means of tracking the vehicle's location using a location information system and providing that information to the owner,

[0784] A communication network means for jointly implementing crime prevention measures by sharing crime prevention information with other users,

[0785] A means of providing owners with information to support safe driving based on past operating data,

[0786] A means of transmitting information to the owner in real time when abnormal vibrations or shocks are detected while driving,

[0787] A system that includes this.

[0788] (Claim 2)

[0789] The system according to claim 1, further comprising means for checking the situation around the vehicle based on data from sensors and cameras and transmitting video to the owner in real time.

[0790] (Claim 3)

[0791] The system according to claim 1, further comprising means for controlling the vehicle's power unit to prevent it from being started illegally using biometric identification technology.

[0792] "Example 2 of combining an emotion engine"

[0793] (Claim 1)

[0794] A means for analyzing information acquired by a detection device mounted on the vehicle to detect abnormalities and emotional states,

[0795] A means of notifying the owner in real time of detected abnormalities and emotional states, and automatically adjusting the in-car environment,

[0796] A means for stopping the vehicle's power unit based on the detection of an abnormality,

[0797] A means of tracking the location of a vehicle using a location determination system and providing that information to the owner,

[0798] A network means that allows users to cooperate in implementing safety measures by sharing safety information with other users,

[0799] A means of providing owners with information to support safe driving based on past driving data,

[0800] A system that includes this.

[0801] (Claim 2)

[0802] The system according to claim 1, further comprising means for checking the conditions around the vehicle based on information from a detection device and transmitting visual information to the owner in real time.

[0803] (Claim 3)

[0804] The system according to claim 1, further comprising means for controlling the vehicle's power unit to prevent unauthorized starting by biometric authentication.

[0805] "Application example 2 when combining with an emotional engine"

[0806] (Claim 1)

[0807] A means for detecting anomalies by analyzing data acquired by detectors mounted on the vehicle,

[0808] A means of notifying the owner of detected anomalies in real time,

[0809] A means for stopping the vehicle's power unit based on the detection of an abnormality,

[0810] A means of tracking the vehicle's location using a global positioning system and providing that information to the owner,

[0811] A communication method for jointly implementing crime prevention measures by sharing crime prevention information with other users,

[0812] A means of providing owners with information to support safe driving based on their past driving history,

[0813] A means for analyzing the emotional state of users using an emotion analysis device inside the vehicle and automatically adjusting the in-vehicle environment according to that state,

[0814] A means of providing feedback and advice based on the user's emotional state,

[0815] A system that includes this.

[0816] (Claim 2)

[0817] The system according to claim 1, further comprising means for checking the situation around the vehicle based on data from a detector and a camera, and transmitting video to the owner in real time.

[0818] (Claim 3)

[0819] The system according to claim 1, further comprising means for controlling the vehicle's power unit to prevent unauthorized starting by biometric authentication. [Explanation of Symbols]

[0820] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for detecting anomalies by analyzing data acquired by detectors and image sensors mounted on the vehicle, A means of notifying the owner's device in real time of detected anomalies, A means for stopping the vehicle's power system based on the detection of an abnormality, A means of tracking the vehicle's location using a location information system and providing that information to the owner, A communication network means for jointly implementing crime prevention measures by sharing crime prevention information with other users, A means of providing owners with information to support safe driving based on past operating data, A means of transmitting information to the owner in real time when abnormal vibrations or shocks are detected while driving, A system that includes this.

2. The system according to claim 1, further comprising means for checking the situation around the vehicle based on data from sensors and cameras and transmitting video to the owner in real time.

3. The system according to claim 1, further comprising means for controlling the vehicle's power unit to prevent it from being started illegally using biometric identification technology.

Images