System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting historical operating information from a database; means for training a AI model for anomaly detection based on the collected information; means for receiving real-time operating information and detecting anomalies using the AI model; means for generating remedies for the detected anomalies; and means for sending the generated anomaly notifications and remedies to user terminals.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In aviation operations, it is important to mitigate accidents and abnormal situations caused by human error. However, managing on-time operations, weather forecasting, and responding to emergencies places a heavy burden on people, and errors are particularly likely to occur when instant decisions are required. Furthermore, with the labor shortage and training challenges that have arisen since the COVID-19 pandemic, measures that do not rely on human resources are required. The purpose of this invention is to solve these issues in aviation operations and achieve safer and more efficient flight management. [Means for solving the problem]

[0005] The present invention includes a means for collecting past operational data from a database and a means for training an AI model for anomaly detection based on the collected data. It also includes a means for receiving operational data in real time and detecting anomalies using the AI ​​model based on that data. It also includes a means for generating countermeasures for detected anomalies and sending the generated anomaly notification and countermeasures to a user terminal. This reduces human error and supports safe and rapid response.

[0006] Furthermore, by adding a means to update the anomaly detection results and countermeasures based on information entered by the user and a means to notify the user terminal of the updated information again, it becomes possible to grasp the situation in real time and take appropriate action. Also, by providing a means to record user operations and the details of the countermeasures taken and save the records in a database, it is possible to display the results in a format that can be reviewed later. This allows for quick post-incident analysis and evaluation of countermeasures.

[0007] A "database" is a collection of information that stores past operational data, weather information, accident and incident history, etc., and can be retrieved and used as needed.

[0008] "Operational Data" means information related to in-flight and ground operations, such as aircraft flight records, location, speed, altitude, and engine status.

[0009] An "AI model" is an algorithm or computational method that is trained based on collected data and used to detect anomalies and make predictions.

[0010] "Training" is the process of feeding collected data into an AI model so that the model learns and improves upon patterns in the data.

[0011] "Real-time data" refers to data that is constantly updated with the current status of the process and is processed and analyzed immediately.

[0012] "Anomaly detection" is the process of using AI models to discover and identify deviations from normal operating conditions.

[0013] "Countermeasures" are specific methods and procedures for proposing and implementing appropriate actions or measures in response to detected abnormalities.

[0014] A "user terminal" is an electronic device used by air traffic controllers and pilots engaged in aviation operations to receive monitoring results and abnormality notifications.

[0015] "Notification" refers to the act of sending information about detected abnormalities and countermeasures to a user terminal to notify the user.

[0016] "Recording" refers to the process of capturing and storing the operations performed by users and the details of the countermeasures implemented in a database.

[0017] "Review" refers to the act of checking and analyzing data and information recorded in the past at a later date.

[0018] "Real-time situational awareness" means being able to instantly check the current operational status and immediately decide and implement countermeasures. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to an air navigation support system, which utilizes AI to support monitoring, guidance, and decision-making in air operations, with the aim of reducing human error and improving the safety and efficiency of air operations.

[0041] System configuration

[0042] The system includes the following major components:

[0043] 1. Database: Stores past operation data, weather information, accident and incident history, etc.

[0044] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[0045] 3. Server: Collects, processes and analyzes data.

[0046] 4. User terminal: Electronic equipment used by controllers and pilots to receive monitoring results and abnormality notifications.

[0047] Program processing

[0048] server

[0049] 1. Data Collection:

[0050] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents.

[0051] 2. Training the AI ​​model:

[0052] The server trains an AI model based on the collected data, which learns patterns in the data to detect and predict anomalies.

[0053] 3. Real-time data reception and analysis:

[0054] The server receives real-time operational data, such as current weather information, flight status, and surveillance camera footage, and provides this data to the AI ​​model, which monitors and analyzes it for any abnormalities.

[0055] 4. Anomaly detection and notification:

[0056] When an anomaly is detected by the AI ​​model, the server analyzes the details and generates a countermeasure, then sends the generated anomaly notification and countermeasure to the user's device.

[0057] Terminal

[0058] 1. Data display:

[0059] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[0060] 2. Receiving user input:

[0061] The terminal receives additional information and judgment results input by the user and transmits them to the server, thereby updating the information in real time.

[0062] 3. Warning Notice:

[0063] The terminal notifies the user of warnings and instructions from the server by voice or visually, and records whether the user acknowledges the warning or not, and sends the record to the server.

[0064] 4. Data recording and archiving:

[0065] The device automatically records the user's actions and the countermeasures taken, sends the data to a server, stores it in a database, and displays it in a format that can be reviewed later.

[0066] User

[0067] 1. Data verification and judgment:

[0068] The user checks the information displayed on the device (for example, an abnormality notification or recommended countermeasures). Based on the displayed information, the user decides on the most appropriate countermeasure in light of the actual situation.

[0069] 2. Enter and execute instructions:

[0070] The user inputs the response they have decided on into the terminal and executes the instructions. By inputting additional information and comments and sending them to the server, more precise responses can be made.

[0071] 3. Warning confirmation and response:

[0072] The user checks the warnings and instructions (for example, an anomaly detection notification) from the terminal and takes the necessary action. The results of the action and the details of the action are entered into the terminal and sent to the server.

[0073] 4. Regular checks:

[0074] Users regularly check the monitoring data and warning information displayed on their devices. Even if no abnormalities are detected, they can take appropriate precautions and check for countermeasures according to the situation.

[0075] Specific examples

[0076] Server operation example

[0077] For example, when monitoring for bad weather, the server collects data on bad weather over the past 10 years and uses an AI model to learn what weather conditions will affect flights. When the server analyzes weather information in real time and detects an approaching storm, it determines whether a particular flight is safe and sends the result to the user's device.

[0078] Example of terminal operation

[0079] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the images transmitted in real time and notifies the user visually and audibly of runway intrusion warnings from the server. After the user performs a confirmation action, the information is sent to the server.

[0080] Examples of user responses

[0081] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight or choose an alternative route, and input their response into their device. The information is then sent to the server and shared immediately with relevant parties. Similarly, a user who receives a false runway intrusion alert can quickly implement a response, input the results into their device, and send them to the server.

[0082] As described above, the system of the present invention utilizes AI to support rapid and appropriate responses while minimizing human error in order to improve the safety and efficiency of operations.

[0083] The processing flow will be explained below.

[0084] server

[0085] Step 1: Data collection

[0086] 1. The server collects historical flight data from a database of flight operations.

[0087] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model.

[0088] Step 2: Training the AI ​​model

[0089] 1. The server feeds the prepared dataset to the AI ​​model and trains the model for anomaly detection and prediction.

[0090] 2. The server evaluates the trained AI model and checks and adjusts its accuracy and processing power.

[0091] Step 3: Receive and analyze real-time data

[0092] 1. The server receives operational data (flight information, aircraft status, weather information, etc.) in real time.

[0093] 2. The server provides the received real-time data to the AI ​​model and analyzes the current operating conditions.

[0094] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation.

[0095] Step 4: Detect anomalies and generate countermeasures

[0096] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[0097] 2. The server generates countermeasures for the detected anomalies.

[0098] Step 5: Sending notification and countermeasures

[0099] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[0100] Terminal

[0101] Step 1: Data display

[0102] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[0103] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[0104] Step 2: Notification of warnings

[0105] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[0106] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[0107] Step 3: Receiving input from the user

[0108] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[0109] Step 4: Record and archive data

[0110] 1. The device automatically records user operations and the countermeasures taken.

[0111] 2. The device sends the recorded data to the server and stores it in a database.

[0112] User

[0113] Step 1: Check the data and assess the situation

[0114] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[0115] 2. Based on the displayed information, the user decides on the best course of action to take in light of the actual operating conditions.

[0116] Step 2: Enter and execute instructions

[0117] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[0118] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[0119] Step 3: Check for warnings and take action

[0120] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[0121] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[0122] Step 4: Regularly check and update

[0123] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[0124] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] To improve the safety and efficiency of air travel, there is a need for real-time anomaly detection and rapid response while minimizing human error. However, conventional systems lack the accuracy to process large amounts of operational data and detect anomalies, making it difficult to respond in real time. This can lead to reduced safety and operational delays, so a new system is needed to solve these issues.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for collecting past flight data from a database, means for training a machine learning model for anomaly detection based on the collected data, means for receiving flight-related data in real time and detecting anomalies using the machine learning model, means for generating countermeasures for the detected anomalies, and means for transmitting the generated anomaly notification and countermeasures to the operation terminal, thereby enabling real-time anomaly detection and rapid response, thereby improving the safety and efficiency of airline operations.

[0130] A "database" is a system designed for the purpose of organizing and storing information, and centrally manages past operational data, weather information, accident and incident history, and more.

[0131] A "machine learning model" is an algorithm that learns patterns based on large amounts of data and makes predictions and detects anomalies in new data.

[0132] "Flight-related data" refers to all information related to aircraft operations, including real-time flight status, weather information, and surveillance camera footage.

[0133] "Anomaly detection" is the process of identifying abnormal situations that deviate from normal operating conditions, allowing for prompt countermeasures to be taken.

[0134] "Countermeasures" refer to actions or measures taken in response to detected abnormalities, with the aim of solving problems or avoiding risks.

[0135] An "operation terminal" is an electronic device used by air traffic controllers and pilots, which notifies users of abnormalities, displays countermeasures, and receives user input.

[0136] A "communication device" is hardware and software for transmitting and receiving data, and is used to exchange real-time data between a server and a terminal.

[0137] "Data Processing Device" means a collection of hardware and software used to process and analyze collected data for anomaly detection and countermeasure generation.

[0138] This invention is a system for improving the safety and efficiency of airline operations, and aims to reduce human error by using machine learning. This system is built through collaboration between servers, terminals, and users, and enables real-time anomaly detection and response.

[0139] server

[0140] The server collects and processes large amounts of operational data, trains machine learning models, and receives and analyzes real-time data. Specifically, it uses the following hardware and software:

[0141] Hardware: High-performance server

[0142] Software: MySQL (database management), TensorFlow (building and training machine learning models), Kafka (receiving and processing real-time data)

[0143] The server collects past operation data from a MySQL database and uses the collected data to train a machine learning model using TensorFlow. The operation data received in real time is processed using Kafka to detect anomalies. If an anomaly is detected, the details are analyzed and a countermeasure is generated. This is all done automatically, and the generated anomaly notification and countermeasure are sent to the terminal.

[0144] For example, to train a forecasting model for severe weather, the server uses TensorFlow to learn how weather conditions affect flights based on 10 years of weather data. If a storm is approaching, the server immediately determines whether a particular flight is safe and notifies the device of the result.

[0145] Terminal

[0146] The terminal visually displays the data sent from the server and supports the user. The following hardware and software are used:

[0147] Hardware: High-resolution display, speakers

[0148] Software: HTML / CSS (dashboard display), JavaScript (asynchronous communication)

[0149] The terminal immediately displays the abnormality notification and countermeasures sent from the server and gives a voice notification. Information entered by the user and the judgment results are sent to the server in real time using JavaScript. The terminal also records operation logs, sends them to the server and stores them in a database.

[0150] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the surveillance video transmitted in real time and notifies the user of runway intrusion warnings visually and audibly. After the user performs a confirmation action, the information is sent to the server.

[0151] User

[0152] The user can make a decision based on the information displayed on the device and take the necessary action. Specific examples of operations are as follows:

[0153] The user checks the abnormality notification displayed on the terminal and decides on the appropriate countermeasure depending on the situation. The user inputs the countermeasure into the terminal and executes it. If necessary, the user can also input additional information or comments and send them to the server in real time.

[0154] For example, if a user sees information about an approaching storm, they can decide whether to delay, cancel, or reroute their flight and input their response plan into their device. The information is then sent to the server and shared with relevant parties immediately. If a false runway intrusion alarm is received, the user can quickly take action and input the results into their device and send them to the server.

[0155] Prompt Sentence Examples

[0156] "Learn how storms affect your flight using weather data from the past decade."

[0157] "Please create a system that provides visual and audio warnings for runway intrusions."

[0158] "Explain how flights will be delayed, canceled, or rerouted based on approaching storms."

[0159] This system significantly improves the safety and efficiency of air travel by linking servers, terminals, and users to detect and respond to abnormalities in real time.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] The server collects historical flight data from the database. Specifically, it connects to the MySQL database and executes the SQL query "SELECT FROM flight_data WHERE date BETWEEN '2010-01-01' AND '2020-12-31';". The input is the SQL query, and the output is data on past flight records, weather information, and accidents and incidents. This output data is used as training data to build an anomaly detection model.

[0163] Step 2:

[0164] The server trains a machine learning model based on the collected data. Data preprocessing involves filling in missing values ​​and normalizing the data. Specifically, the model is trained using TensorFlow using "model.fit(training_data, labels, epochs=50)". The input is the preprocessed data, and the output is a trained anomaly detection model.

[0165] Step 3:

[0166] The server receives flight-related data in real time. Specifically, it uses Kafka to receive data via "consumer.poll(100)" and processes it as real-time data. The input is real-time flight status, weather information, and surveillance camera footage, and the output is packets containing this data.

[0167] Step 4:

[0168] The server inputs real-time data into a machine learning model for analysis. Specifically, it inputs the received data into "model.predict(new_flight_data)" to detect anomalies. The input is real-time data, and the output is the prediction result (presence or absence of anomalies).

[0169] Step 5:

[0170] When an anomaly is detected, the server analyzes the details and generates countermeasures. Specifically, it runs "analyze_anomaly(predictions)" to analyze the details of the anomaly and generate countermeasures. The input is the prediction result, and the output is an anomaly notification including countermeasures.

[0171] Step 6:

[0172] The server sends the generated abnormality notification and countermeasures to the operation terminal. Specifically, it uses Kafka to send notifications as follows: "producer.send('alerts', alert_message)". The input is the abnormality notification and countermeasures, and the output is a notification message to the operation terminal.

[0173] Step 7:

[0174] The terminal displays the data sent from the server on the screen. Specifically, a warning message is displayed on a dashboard created with HTML / CSS using "document.getElementById('alert').innerHTML = alert_message;". The input is the notification message from the server, and the output is the abnormality notification and countermeasures displayed on the terminal screen.

[0175] Step 8:

[0176] The device receives additional information and decision results entered by the user and sends them to the server via asynchronous communication. Specifically, it uses the JavaScript "fetch" API, using "fetch(' / submit-decisions', { method: 'POST', body: JSON.stringify(user_decision)})". The input is the user's decision result, and the output is the data to be sent to the server.

[0177] Step 9:

[0178] The terminal notifies the user of warnings and instructions from the server visually and audibly. Specifically, it uses the browser's notification API to display a warning with "new Notification('Warning', { body: 'An abnormality has been detected on the runway.'})". The input is the warning message from the server, and the output is the notification to the user.

[0179] Step 10:

[0180] The user makes a decision based on the information displayed on the terminal and takes the necessary action. Specifically, the user checks the warning message and inputs the action, such as flight delay, cancellation, or rerouting, into the terminal. The input is the information displayed on the terminal, and the output is the action taken and its results.

[0181] Step 11:

[0182] User operations and the details of the countermeasures taken are recorded on the terminal, and the data is sent to the server and stored in a database. Specifically, the operation log is collected in JSON format and sent to the server using "fetch(' / log-actions', { method: 'POST', body: JSON.stringify(action_log)})". The input is the user operation log, and the output is the recorded data stored in the database.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Conventional autonomous vehicle systems are inadequate in their real-time response to changes in traffic and weather conditions, making it difficult to detect abnormalities and generate appropriate countermeasures. This raises concerns about traffic accidents and reduced safety. Furthermore, they lack the ability to quickly incorporate user feedback and improve system accuracy. There is a need for a system that can solve these issues and improve the safety and efficiency of autonomous vehicles.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes: means for collecting past operation data from a database; means for training an AI model for anomaly detection based on the collected data; means for receiving operation data in real time and detecting anomalies using the AI ​​model; means for generating countermeasures for the detected anomalies; means for transmitting the generated anomaly notification and countermeasures to a user terminal; means for collecting real-time sensor data and weather information from the autonomous vehicle; means for analyzing the collected data using the AI ​​model to detect abnormal conditions; means for displaying the anomaly notification and countermeasures visually and audibly on the user terminal; means for receiving user input and providing real-time feedback to the system; means for recording user operations and anomaly response details and saving the records in a database; and means for displaying the saved record data in a format that can be reviewed later. This makes it possible to monitor changes in traffic and weather conditions in real time while the autonomous vehicle is operating, quickly detect abnormal conditions, and automatically generate appropriate countermeasures, thereby improving safety and efficiency.

[0188] A "database" is a storage device that accumulates and stores past operation data, weather information, traffic accident data, and the like.

[0189] "Operation data" refers to all information that indicates the vehicle's operating status, such as various sensor data, camera footage, location information, and speed information from an autonomous vehicle.

[0190] An "AI model" is an algorithm trained to detect and predict anomalies based on collected data.

[0191] "Sensor data" refers to real-time operational information acquired by various sensors (e.g., cameras, lidar, radar) installed in autonomous vehicles.

[0192] "Weather information" refers to information about weather conditions during operation, such as wind speed, rainfall, and temperature.

[0193] An "abnormal state" is an unusual situation detected during operation (e.g., sudden deceleration, lane departure, approaching obstacle, etc.).

[0194] "Countermeasures" are specific actions or measures that should be taken in response to a detected abnormal condition.

[0195] A "user terminal" is a device (for example, an in-car display or a smartphone) that displays information sent from a server and receives user input.

[0196] "Feedback" is real-time input, such as additional information or decisions provided by the user, that is used to improve the system.

[0197] "Voice notification" is a means of conveying warnings and instructions to the user by voice.

[0198] A "visual display" is a means of displaying information using text and graphics on a user terminal display.

[0199] This invention is a system that improves the operational safety and efficiency of autonomous vehicles. It uses AI to monitor changes in traffic and weather conditions in real time, quickly detects abnormal conditions, and automatically generates appropriate countermeasures.

[0200] System Configuration

[0201] The system includes the following major components:

[0202] server

[0203] The server has the following roles:

[0204] 1. Database (accumulation of past operation data):

[0205] The database will store past driving data, weather information, traffic accident data, etc. This data is collected from autonomous vehicles in operation.

[0206] 2. Training the AI ​​model:

[0207] The server uses the collected data to train an AI model, which learns patterns of abnormal conditions and enables real-time anomaly detection.

[0208] 3. Real-time data reception and analysis:

[0209] The server receives sensor data and weather information sent from the autonomous vehicle in real time and analyzes it using an AI model.

[0210] 4. Detect anomalies and generate countermeasures:

[0211] If an anomaly is detected, the details are analyzed and countermeasures are generated, providing specific measures to keep the autonomous vehicle operating safely.

[0212] 5. Sending abnormality notification and countermeasures to user devices:

[0213] A notification of the detected abnormality and the generated countermeasure are sent to a user terminal inside the vehicle.

[0214] User devices (displays and smartphones inside autonomous vehicles)

[0215] The user terminal provides the following functions:

[0216] 1. Data display:

[0217] Displays abnormal notifications and countermeasures sent from the server visually and audibly.

[0218] 2. Receiving user input:

[0219] Receive feedback and additional information from the user and send that information to the server.

[0220] 3. Warning notice:

[0221] If an abnormality is detected, the user will be notified with a visual and audio warning.

[0222] 4. Data recording and archiving:

[0223] The user's operations and responses are recorded and the data is saved in a database.

[0224] User

[0225] The user roles are as follows:

[0226] 1. Check the data and take action:

[0227] Check for abnormality notifications and recommended actions, and take action as necessary.

[0228] 2. Providing Feedback:

[0229] Provide real-time feedback to the system to improve data accuracy.

[0230] Specific examples

[0231] For example, if an autonomous vehicle detects sudden deceleration while traveling on a highway, the system operates as follows:

[0232] 1. The server analyzes sudden deceleration data on the highway using an AI model and detects it as an abnormal condition.

[0233] 2. The server generates a countermeasure for the detected abnormal condition, such as "activate the automatic brakes to prevent deceleration."

[0234] 3. The generated abnormality notification and countermeasures are sent to the user terminal.

[0235] 4. The user device visually displays the abnormal condition and countermeasures on the in-car display and also notifies the driver audibly.

[0236] 5. The user acknowledges the notification and provides appropriate feedback to the system (e.g., "Got it, brake on").

[0237] 6. User actions are recorded and saved in a format that can be reviewed at a later date.

[0238] Example prompt sentence:

[0239] For example, if an AI model detects a sudden deceleration on a highway and generates a countermeasure, it might generate a prompt like this:

[0240] Abnormal condition: sudden deceleration

[0241] Location information: Expressway X section

[0242] Recommended action: Activate the automatic brakes to prevent slowing down.

[0243] Notification content: Sudden deceleration detected on X section of highway. Activate automatic brakes to prevent deceleration.

[0244] This will increase the safety and efficiency of self-driving vehicles.

[0245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0246] Step 1:

[0247] The server collects historical driving data, weather information, traffic accident data, etc. from a database. The input is the database, and the output is raw data used to train the AI ​​model. Specifically, it uses SQL queries to retrieve the required data and saves it in CSV or JSON format.

[0248] Step 2:

[0249] The server trains an AI model based on the collected data. The input is the raw data collected in step 1, and the output is the trained AI model. Specifically, an anomaly detection model is built and trained using Python libraries such as Scikit-learn and TensorFlow.

[0250] Step 3:

[0251] The server receives sensor data and weather information sent from autonomous vehicles in real time. The input is real-time data, and the output is a dataset to be input into the AI ​​model. Specifically, the server receives real-time data via WebSocket or HTTP, preprocesses it, and formats it into a form that can be provided to the AI ​​model.

[0252] Step 4:

[0253] The server inputs the received real-time data into the AI ​​model for analysis. The input is a formatted data set, and the output is the analysis result (presence or absence of anomalies). Specifically, the predict method of the AI ​​model is called to detect abnormalities.

[0254] Step 5:

[0255] When the server detects an anomaly, it analyzes the details and generates a countermeasure. The input is the analysis result, and the output is a specific countermeasure and an anomaly notification message. Specifically, the generated anomaly notification and countermeasure are converted into JSON format and sent to the user's device.

[0256] Step 6:

[0257] The terminal displays the abnormality notification and countermeasures sent from the server visually and audibly. The input is the message from the server, and the output is the notification to the user. Specifically, the abnormality notification and countermeasures are displayed on the screen, and audio guidance is provided through the speaker.

[0258] Step 7:

[0259] The user checks the notified abnormal condition and takes appropriate action. The input is the notification from the terminal, and the output is the action taken and its results. Specifically, the user inputs feedback such as "Got it, apply brakes" into the terminal, which actually applies the brakes.

[0260] Step 8:

[0261] The device records the user's operations and the countermeasures taken, and sends the records to the server. The input is the user's feedback and operations, and the output is the recorded data. Specifically, the operations are saved in a log file and periodically uploaded to the server.

[0262] Step 9:

[0263] The server stores the recorded data in a database and displays it in a format that can be reviewed later. The input is the recorded data, and the output is the display items that can be reviewed by the user. Specifically, it inserts data into a database and visualizes the data in a web application or dashboard.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[0266] System configuration

[0267] The system includes the following major components:

[0268] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[0269] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[0270] 3. Server: Collects, processes and analyzes data.

[0271] 4. User terminal: An electronic device used by controllers and pilots to receive monitoring results and abnormality notifications.

[0272] 5. Emotion Engine: An engine that recognizes user emotions in real time and analyzes the data.

[0273] Program processing

[0274] server

[0275] 1. Data Collection:

[0276] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents, as well as user sentiment data.

[0277] 2. Training the AI ​​model:

[0278] The server trains an AI model based on the collected data. The AI ​​model learns patterns in the data to detect and predict anomalies. In particular, by including user emotional data, it becomes possible to provide emotion-based countermeasures for anomalies.

[0279] 3. Real-time data reception and analysis:

[0280] The server receives real-time operational data (flight information, aircraft status, weather information, etc.), provides this data to the AI ​​model, and analyzes the current operational status. The analysis results are used to monitor whether there are any operational anomalies.

[0281] 4. Detect anomalies and generate countermeasures:

[0282] The server uses an AI model to detect anomalies and identify their details (location, scope of impact, cause, etc.). It then generates countermeasures for the detected anomalies. Based on information from the emotion engine, it adjusts the priority of countermeasures according to the user's emotional state.

[0283] 5. Notification of abnormalities and response measures:

[0284] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[0285] Terminal

[0286] 1. Data display:

[0287] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[0288] 2. Warning Notice:

[0289] The terminal notifies the user of warnings and instructions from the server audibly and visually, records whether the user acknowledges the warning, and sends that information to the server.

[0290] 3. Receiving user input:

[0291] The terminal receives additional information and judgment results input by the user and transmits the data to the server in real time.

[0292] 4. Collecting Emotional Data:

[0293] The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[0294] 5. Data recording and archiving:

[0295] The device automatically records the user's operations and the countermeasures taken, and sends the recorded data to a server where it is stored in a database.

[0296] User

[0297] 1. Data review and situation assessment:

[0298] The user checks the information displayed on the device (such as abnormality notifications and countermeasures). Based on the displayed information, the user decides on the optimal countermeasure taking into account the actual operating conditions. The user can also refer to their own emotional data collected by the emotion engine.

[0299] 2. Enter and execute instructions:

[0300] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction. The details and results of the executed countermeasure are also input into the terminal, and the data is sent to the server.

[0301] 3. Warning confirmation and response:

[0302] The user checks the warnings and instructions sent from the device and takes the necessary action promptly. The device records the action and the situation, and sends that information to the server.

[0303] 4. Regular review and updates:

[0304] Users regularly check the monitoring data and warning information displayed on their devices, and take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0305] Specific examples

[0306] Server operation example

[0307] For example, when monitoring for abnormalities caused by bad weather, the server collects and learns from past bad weather data and analyzes it using an AI model. In addition, it also collects and learns data on the user's emotional state during past bad weather. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the results to the user's device. At this point, it also presents countermeasures prioritized according to the user's emotional state.

[0308] Example of terminal operation

[0309] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly, and the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[0310] Examples of user responses

[0311] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can refer to the analysis results of the emotion engine to select the most appropriate response. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, a user who receives an alert about a false intrusion on the runway can quickly implement a response and enter the results into the device. The emotion engine will evaluate the user's emotional state, and the results will also be used to improve the service.

[0312] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[0313] The processing flow will be explained below.

[0314] server

[0315] Step 1: Data collection

[0316] 1. The server collects past flight data, weather information, accident and incident history, and past emotion data from a database related to flight operations.

[0317] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model and emotion engine.

[0318] Step 2: Training the AI ​​model and emotion engine

[0319] 1. The server feeds the prepared dataset to the AI ​​model and emotion engine to train it for anomaly detection and emotion recognition.

[0320] 2. The server evaluates the trained AI model and emotion engine, and checks and adjusts its accuracy and processing power.

[0321] Step 3: Receive and analyze real-time data

[0322] 1. The server receives real-time operational data (flight information, aircraft status, weather information, etc.) and user emotion data.

[0323] 2. The server provides the received real-time data to the AI ​​model and emotion engine to analyze the current driving situation and the user's emotional state.

[0324] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation and monitors the user's emotional state.

[0325] Step 4: Detect anomalies and generate countermeasures

[0326] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[0327] 2. The server uses an emotion engine to evaluate the user's emotional state and reflects this in the anomaly detection results.

[0328] 3. The server generates countermeasures for the detected anomalies and adjusts the priority of the countermeasures according to the user's emotional state.

[0329] Step 5: Sending notification of abnormalities and countermeasures

[0330] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[0331] Terminal

[0332] Step 1: Data display

[0333] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[0334] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[0335] Step 2: Notification of warnings

[0336] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[0337] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[0338] Step 3: Receiving User Input

[0339] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[0340] Step 4: Collecting emotion data

[0341] 1. The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[0342] Step 5: Record and archive data

[0343] 1. The device automatically records user operations and the countermeasures taken.

[0344] 2. The terminal periodically sends the recorded data to the server and stores it in a database.

[0345] User

[0346] Step 1: Check the data and assess the situation

[0347] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[0348] 2. Based on the displayed information, the user decides on the best course of action to take, taking into account the actual driving situation and their own emotional state.

[0349] Step 2: Enter and execute instructions

[0350] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[0351] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[0352] Step 3: Check for warnings and take action

[0353] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[0354] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[0355] Step 4: Regularly check and update

[0356] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[0357] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0358] Specific examples

[0359] Server operation example

[0360] For example, when monitoring for abnormalities caused by bad weather, the server collects past bad weather data and analyzes it using an AI model. It also collects data on the user's emotional state during bad weather and analyzes it using an emotion engine. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of a specific flight and sends the results to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[0361] Example of terminal operation

[0362] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly. At this time, the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, along with the analyzed emotion data.

[0363] Examples of user responses

[0364] For example, a user who has received information about an approaching storm may decide to delay or cancel their flight, or to choose an alternative route. In this case, the user may refer to the analysis results of the emotion engine to select the optimal response. The selected response is then entered into the device, and the information is sent to the server and shared with relevant parties immediately. Furthermore, the user's emotional state is also recorded and analyzed, and this information is used to improve the service.

[0365] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[0366] Example 2

[0367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] Conventional systems for detecting anomalies and providing countermeasures in aviation operations are prone to human error and place a heavy psychological burden on users. In particular, in emergency situations, the user's emotional state is not properly taken into account, which can result in the selection of the most appropriate countermeasure. This has led to a demand for improved safety and efficiency.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0370] In this invention, the server includes means for collecting past operation data and emotion data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures based on the detected anomalies and the user's emotional state, and means for transmitting the generated anomaly notification and countermeasures to the user terminal. This makes it possible to provide anomaly countermeasures that take the user's emotional state into consideration, thereby achieving improved safety and efficiency.

[0371] The "database" is an information collection point for storing and managing various data required by the system, such as past operation data and emotion data.

[0372] "Emotion data" is information that indicates the user's psychological state, and is data extracted from facial expressions, tone of voice, behavior, and the like.

[0373] An "AI model" is an algorithm that is trained based on collected data to detect anomalies and make predictions.

[0374] "Server" means a central control unit that collects, processes, and analyzes data and manages communications with user terminals.

[0375] A "user terminal" is an electronic device used by air traffic controllers and pilots to receive information and implement countermeasures.

[0376] "Anomaly detection" is the process of using AI models to analyze operational data and identify abnormal conditions or malfunctions.

[0377] "Countermeasures" refer to specific actions or instructions to be taken in response to detected abnormalities.

[0378] The "emotion engine" is software that analyzes the user's facial expressions and tone of voice, and collects and evaluates emotional data in real time.

[0379] An "abnormality notification" is a message or notification that notifies a user terminal of a detected abnormality and a countermeasure.

[0380] "Users" are human operators such as controllers and pilots who operate the system, review information, and implement countermeasures.

[0381] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[0382] The system includes the following major components:

[0383] 1. Database

[0384] It stores past operation data, weather information, accident and incident history, user emotion data, etc. To do this, it uses a commonly used relational database (e.g., MySQL).

[0385] 2. AI Model

[0386] This is an algorithm trained on database information to perform anomaly detection and prediction. It uses the Python TensorFlow library. In particular, by including user emotional data, it is possible to provide emotion-based countermeasures for anomalies.

[0387] 3. Server

[0388] It collects, processes, and analyzes data. It receives operational data (flight information, aircraft status, weather information, etc.) in real time and provides it to an AI model to analyze operational status. The server receives real-time data using WebSocket technology and sends the analysis results to the user's device.

[0389] 4. User Device

[0390] This is an electronic device used by air traffic controllers and pilots to receive monitoring results and abnormality notifications. The terminal displays information sent from the server in real time and issues audio and visual warnings. It also sends additional information and judgment results entered by the user to the server in real time, and uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data.

[0391] 5. Emotion Engine

[0392] This is an engine that recognizes user emotions in real time and analyzes the data. The emotion engine uses a camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. The collected data is sent to a server and used to generate countermeasures against abnormalities.

[0393] Specific examples

[0394] Server operation example

[0395] For example, when a server monitors for abnormalities caused by bad weather, it collects and learns from past bad weather data and analyzes it using an AI model. It also collects and learns data on the user's emotional state during past bad weather. If it analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the result to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[0396] Example of terminal operation

[0397] For example, if a device monitors a runway using a surveillance camera, it will display the video in real time and notify the user visually and audibly of any runway intrusion warnings from the server. At this time, the emotion engine analyzes the user's facial expressions and tone of voice, and when the user acknowledges the warning, that information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[0398] Examples of user responses

[0399] For example, if a user sees information about an approaching storm, they can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can select the optimal response, taking into account the analysis results of the emotion engine. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, if a user receives an alert about a false runway intrusion, they can quickly implement a response and enter the results into the device. The emotion engine will then evaluate the user's emotional state, and the results will also be used to improve the service.

[0400] Prompt Sentence Examples

[0401] "Please explain how the server collects historical severe weather data and operational data to train the AI ​​model. Also, please describe the process by which the weather information is analyzed in real time and anomaly response measures are generated based on the user's emotional state."

[0402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0403] Program processing flow

[0404] server

[0405] Step 1:

[0406] Data collection

[0407] The server connects to the flight database to retrieve flight records, weather information, past accident and incident data, and user sentiment data. These data are retrieved using a MySQL database.

[0408] Input: Database connection information

[0409] Processing: Extract the required data from the database using SQL queries.

[0410] Output: A set of operational and emotional data

[0411] Step 2:

[0412] Training an AI model

[0413] The server uses the Python TensorFlow library to train the AI ​​model, which learns patterns to detect anomalies based on collected operational and emotional data.

[0414] Input: A set of operational data and emotion data

[0415] Processing: After data preprocessing (normalization, feature extraction, etc.), training is performed using an AI model.

[0416] Output: A trained AI model

[0417] Step 3:

[0418] Receiving and analyzing real-time data

[0419] The server periodically receives real-time data (flight information, aircraft status, weather data, etc.) sent from the aircraft using WebSocket technology, and provides the received data to an AI model to analyze the current operating status.

[0420] Input: Real-time data

[0421] Processing: The received data is fed into an AI model for analysis to determine whether there are any anomalies.

[0422] Output: Current operating status and abnormality detection results

[0423] Step 4:

[0424] Detecting anomalies and generating countermeasures

[0425] The server uses an AI model to detect anomalies. If an anomaly is detected, it identifies the details (location of occurrence, scope of impact, cause, etc.), adjusts the priority of countermeasures based on data from the emotion engine, and then generates countermeasures.

[0426] Input: Anomaly detection results and emotion data

[0427] Action: Generate and prioritize countermeasures based on detected anomalies.

[0428] Output: Abnormal notification and countermeasures

[0429] Step 5:

[0430] Sending abnormality notifications and countermeasures

[0431] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[0432] Input: Abnormal notification and countermeasures

[0433] Processing: Convert the notification data into an appropriate format and send it to the user terminal.

[0434] Output: Abnormal notification to user terminal

[0435] Terminal

[0436] Step 1:

[0437] Data Display

[0438] The terminal displays the monitoring results and abnormality notifications sent from the server in real time, using an interface that uses a GUI library.

[0439] Input: Abnormal notification and countermeasures

[0440] Processing: Updates various display elements of the GUI.

[0441] Output: Displayed abnormality notification and countermeasures

[0442] Step 2:

[0443] Warning Notification

[0444] The terminal will notify the user of warnings and instructions from the server audibly and visually, including playing audio files and visual alerts.

[0445] Input: Error notification and warning information

[0446] Action: Plays an audio and displays a visual alert.

[0447] Output: Audio and visual warnings

[0448] Step 3:

[0449] Receiving User Input

[0450] The terminal receives input from the user and transmits the data to the server in real time, either via a touchscreen or keyboard input.

[0451] Input: User-entered data

[0452] Processing: Sends the received user input to the server.

[0453] Output: Send input data to the server

[0454] Step 4:

[0455] Collecting Emotional Data

[0456] The device uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data, using a camera and microphone.

[0457] Input: User's facial expression data and voice data

[0458] Processing: Emotion data is generated using facial expression analysis algorithms and voice analysis algorithms.

[0459] Output: Emotion data

[0460] Step 5:

[0461] Data recording and archiving

[0462] The device automatically records the user's actions and the countermeasures taken, and transmits the data to the server using local storage and network communication.

[0463] Input: User operation log and execution details

[0464] Processing: The log data is stored in a format and sent to the server.

[0465] Output: Send log data to the server

[0466] User

[0467] Step 1:

[0468] Data confirmation and situation assessment

[0469] The user checks the information displayed on the device, determines the best course of action based on the operational status, and takes appropriate action based on the displayed information.

[0470] Input: Abnormality notification and countermeasures displayed on the terminal

[0471] Processing: Analyze the displayed information and make decisions based on the situation.

[0472] Output: Decision results and action selection

[0473] Step 2:

[0474] Entering and executing instructions

[0475] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction.

[0476] Input: Workaround

[0477] Processing: Executes the input instructions and inputs the results to the terminal.

[0478] Output: Input data and execution results to the terminal

[0479] Step 3:

[0480] Checking and responding to warnings

[0481] The user checks the warnings and instructions from the terminal and takes appropriate action promptly.

[0482] Input: Warning notices and instructions

[0483] Action: Assess the situation and take the most appropriate action.

[0484] Output: Countermeasure execution results

[0485] Step 4:

[0486] Regular review and updates

[0487] Users regularly check the monitoring data and warning information displayed on their devices and take preventive measures or make advance preparations as necessary.

[0488] Input: Monitoring data and warning information

[0489] Processing: Regularly review the data and take any necessary action.

[0490] Output: Preventive measures taken and preparatory results

[0491] (Application example 2)

[0492] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0493] Conventional autonomous vehicle traffic management systems can detect anomalies and generate countermeasures in real time, but they cannot consider the emotional state of passengers and drivers. As a result, psychological burden increases and appropriate responses are delayed. Therefore, in order to improve safety and efficiency during operation, it is necessary to analyze the user's emotional state in real time and provide appropriate countermeasures.

[0494] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past operation data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures for the detected anomaly, means for transmitting the generated anomaly notification and countermeasures to the user terminal, means for analyzing the user's emotional state using an emotion engine, and means for adjusting the priority of the countermeasures based on the analyzed emotional state. This makes it possible to provide anomaly countermeasures according to the user's emotional state.

[0495] The "database" is an information management system for storing past operational data, weather information, accident and incident history, and user emotional data.

[0496] "Operation data" refers to information about the operating status of an autonomous vehicle, including, for example, flight records, weather information, and vehicle sensor data.

[0497] An "AI model" is an algorithm that is trained on information from a database to perform anomaly detection and prediction.

[0498] "Anomaly detection" is the process of using AI models to identify unusual behaviors or situations in operational data.

[0499] "Countermeasures" are specific measures or action plans for the detected abnormality, and are intended to ensure safety and improve operational efficiency.

[0500] A "user terminal" is a device used by controllers and drivers, and is an electronic device for receiving monitoring results and abnormality notifications.

[0501] The "emotion engine" is a system that analyzes the user's emotional state in real time and reflects that data in operational management.

[0502] "Priority" is an index that indicates the importance and urgency of implementing an abnormality response measure, and is adjusted based on the emotional state of the user.

[0503] An "abnormality notification" is a message for transmitting information about a detected abnormality to a user terminal.

[0504] "Analysis" is the process of identifying and understanding situations and patterns based on collected data.

[0505] The present invention combines an emotion engine with a safe driving management system for autonomous vehicles, making it possible to provide countermeasures against abnormalities that take into account the emotional state of the user. A specific embodiment of the system for implementing the present invention will be described below.

[0506] System configuration

[0507] The system includes the following major components:

[0508] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[0509] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[0510] 3. Server: Collects, processes and analyzes data.

[0511] 4. User terminal: An electronic device used by the driver or passenger to receive monitoring results and abnormality notifications.

[0512] 5. Emotion engine: Analyzes user emotions in real time and reflects that data in operation management.

[0513] Program processing

[0514] server

[0515] The server collects historical driving data (e.g., speed, location, and surrounding obstacle information) from the driving database. This data includes vehicle sensor data, real-time weather data, traffic condition data, and user emotion data. Based on the collected data, the server trains an AI model to detect and predict anomalies. Furthermore, the server receives real-time driving data, detects anomalies using the AI ​​model, and generates countermeasures. The server adjusts the priority of countermeasures based on the emotion data analyzed by the emotion engine.

[0516] User terminal

[0517] The user device receives the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server and notifies the user visually and audibly. The device also uses a camera and smart glasses to analyze the user's facial expressions and tone of voice to collect emotional data. This allows the user's emotional state to be sent to the server in real time and used to adjust the priority of countermeasures to anomalies.

[0518] User

[0519] The user checks the information displayed on the device (such as abnormality notifications and countermeasures) and selects and executes the most appropriate countermeasure for the situation. In doing so, the user also takes into account their own emotional data analyzed by the emotion engine. For example, after checking information about an approaching storm, the user can decide whether to delay or cancel a flight or select an alternative route. The countermeasures taken are entered into the device and recorded and analyzed on the server.

[0520] Specific examples

[0521] For example, when an autonomous vehicle detects abnormal data during bad weather, the system predicts the abnormality based on past weather data and the user's emotional state at the time. If passenger anxiety increases due to the emotion engine, the system can quickly provide high-priority countermeasures such as delays or route changes. After the user confirms and acts on the warning, the information is sent to a server and stored in a database.

[0522] Example prompts to input to the generative AI model

[0523] Get up-to-date traffic data, calculate safe and efficient routes based on weather data and vehicle sensor data, and take into account the emotional state (e.g., stress, relief) of the passengers and driver in the car.

[0524] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0525] Step 1:

[0526] The server collects historical driving data from a database. This data includes vehicle sensor data (speed, location, obstacle information, etc.), weather data, traffic condition data, and user emotion data. This creates a dataset that provides a comprehensive understanding of driving conditions. The collected data is used to train AI models for anomaly detection and prediction.

[0527] Input: Historical driving data, sensor data, weather data, traffic data, user emotion data

[0528] Output: Collected dataset

[0529] Step 2:

[0530] The server trains the AI ​​model based on the collected data. This training process learns patterns in the data and builds algorithms capable of detecting and predicting anomalies. In particular, by including user emotional data, it is possible to generate countermeasures to anomalies that take into account the user's emotional state.

[0531] Input: Collected dataset

[0532] Output: A trained AI model

[0533] Step 3:

[0534] The server receives real-time operational data, including the vehicle's current sensor data, real-time weather information, and traffic data, which is then fed into an AI model to analyze the current operational situation.

[0535] Input: Real-time operation data, sensor data, weather information, traffic condition data

[0536] Output: Parsed operational status data

[0537] Step 4:

[0538] The server uses an AI model to detect anomalies. It then identifies the details of the detected anomaly (location, scope of impact, cause, etc.) and generates countermeasures. It then adjusts the priority of countermeasures based on the user's emotional data collected by the emotion engine.

[0539] Input: Analyzed driving status data, real-time emotion data

[0540] Output: Coordinated response measures, anomaly notification

[0541] Step 5:

[0542] The server sends the generated abnormality notification and countermeasures to the user terminal. The user terminal receives this and notifies the user visually and audibly. The user checks the information displayed on the terminal and selects and executes the appropriate countermeasures.

[0543] Input: Coordinated action, anomaly notification

[0544] Output: User's action selection information

[0545] Step 6:

[0546] The user device uses a camera and smart glasses to analyze the user's facial expressions and tone of voice, collecting emotional data in real time. This emotional data is sent to a server and used to prioritize countermeasures against abnormalities.

[0547] Input: Real-time video and audio data of the user

[0548] Output: Real-time emotion data

[0549] Step 7:

[0550] The user enters the details and results of the selected action into the device and sends the data to the server, which stores this information in a database and uses it for future analysis and service improvement.

[0551] Input: User-entered details of countermeasures, execution results

[0552] Output: Details of saved countermeasures, execution result data

[0553] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0554] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0555] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0556] [Second embodiment]

[0557] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0558] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0559] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0560] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0561] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0562] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0563] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0564] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0565] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0568] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0569] This invention relates to an air navigation support system, which utilizes AI to support monitoring, guidance, and decision-making in air operations, with the aim of reducing human error and improving the safety and efficiency of air operations.

[0570] System configuration

[0571] The system includes the following major components:

[0572] 1. Database: Stores past operation data, weather information, accident and incident history, etc.

[0573] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[0574] 3. Server: Collects, processes and analyzes data.

[0575] 4. User terminal: Electronic equipment used by controllers and pilots to receive monitoring results and abnormality notifications.

[0576] Program processing

[0577] server

[0578] 1. Data Collection:

[0579] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents.

[0580] 2. Training the AI ​​model:

[0581] The server trains an AI model based on the collected data, which learns patterns in the data to detect and predict anomalies.

[0582] 3. Real-time data reception and analysis:

[0583] The server receives real-time operational data, such as current weather information, flight status, and surveillance camera footage, and provides this data to the AI ​​model, which monitors and analyzes it for any abnormalities.

[0584] 4. Anomaly detection and notification:

[0585] When an anomaly is detected by the AI ​​model, the server analyzes the details and generates a countermeasure, then sends the generated anomaly notification and countermeasure to the user's device.

[0586] Terminal

[0587] 1. Data display:

[0588] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[0589] 2. Receiving user input:

[0590] The terminal receives additional information and judgment results input by the user and transmits them to the server, thereby updating the information in real time.

[0591] 3. Warning Notice:

[0592] The terminal notifies the user of warnings and instructions from the server by voice or visually, and records whether the user acknowledges the warning or not, and sends the record to the server.

[0593] 4. Data recording and archiving:

[0594] The device automatically records the user's actions and the countermeasures taken, sends the data to a server, stores it in a database, and displays it in a format that can be reviewed later.

[0595] User

[0596] 1. Data verification and judgment:

[0597] The user checks the information displayed on the device (for example, an abnormality notification or recommended countermeasures). Based on the displayed information, the user decides on the most appropriate countermeasure in light of the actual situation.

[0598] 2. Enter and execute instructions:

[0599] The user inputs the response they have decided on into the terminal and executes the instructions. By inputting additional information and comments and sending them to the server, more precise responses can be made.

[0600] 3. Warning confirmation and response:

[0601] The user checks the warnings and instructions (for example, an anomaly detection notification) from the terminal and takes the necessary action. The results of the action and the details of the action are entered into the terminal and sent to the server.

[0602] 4. Regular checks:

[0603] Users regularly check the monitoring data and warning information displayed on their devices. Even if no abnormalities are detected, they can take appropriate precautions and check for countermeasures according to the situation.

[0604] Specific examples

[0605] Server operation example

[0606] For example, when monitoring for bad weather, the server collects data on bad weather over the past 10 years and uses an AI model to learn what weather conditions will affect flights. When the server analyzes weather information in real time and detects an approaching storm, it determines whether a particular flight is safe and sends the result to the user's device.

[0607] Example of terminal operation

[0608] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the images transmitted in real time and notifies the user visually and audibly of runway intrusion warnings from the server. After the user performs a confirmation action, the information is sent to the server.

[0609] Examples of user responses

[0610] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight or choose an alternative route, and input their response into their device. The information is then sent to the server and shared immediately with relevant parties. Similarly, a user who receives a false runway intrusion alert can quickly implement a response, input the results into their device, and send them to the server.

[0611] As described above, the system of the present invention utilizes AI to support rapid and appropriate responses while minimizing human error in order to improve the safety and efficiency of operations.

[0612] The processing flow will be explained below.

[0613] server

[0614] Step 1: Data collection

[0615] 1. The server collects historical flight data from a database of flight operations.

[0616] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model.

[0617] Step 2: Training the AI ​​model

[0618] 1. The server feeds the prepared dataset to the AI ​​model and trains the model for anomaly detection and prediction.

[0619] 2. The server evaluates the trained AI model and checks and adjusts its accuracy and processing power.

[0620] Step 3: Receive and analyze real-time data

[0621] 1. The server receives operational data (flight information, aircraft status, weather information, etc.) in real time.

[0622] 2. The server provides the received real-time data to the AI ​​model and analyzes the current operating conditions.

[0623] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation.

[0624] Step 4: Detect anomalies and generate countermeasures

[0625] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[0626] 2. The server generates countermeasures for the detected anomalies.

[0627] Step 5: Sending notification and countermeasures

[0628] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[0629] Terminal

[0630] Step 1: Data display

[0631] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[0632] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[0633] Step 2: Notification of warnings

[0634] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[0635] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[0636] Step 3: Receiving input from the user

[0637] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[0638] Step 4: Record and archive data

[0639] 1. The device automatically records user operations and the countermeasures taken.

[0640] 2. The device sends the recorded data to the server and stores it in a database.

[0641] User

[0642] Step 1: Check the data and assess the situation

[0643] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[0644] 2. Based on the displayed information, the user decides on the best course of action to take in light of the actual operating conditions.

[0645] Step 2: Enter and execute instructions

[0646] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[0647] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[0648] Step 3: Check for warnings and take action

[0649] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[0650] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[0651] Step 4: Regularly check and update

[0652] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[0653] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0654] Example 1

[0655] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] To improve the safety and efficiency of air travel, there is a need for real-time anomaly detection and rapid response while minimizing human error. However, conventional systems lack the accuracy to process large amounts of operational data and detect anomalies, making it difficult to respond in real time. This can lead to reduced safety and operational delays, so a new system is needed to solve these issues.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0658] In this invention, the server includes means for collecting past flight data from a database, means for training a machine learning model for anomaly detection based on the collected data, means for receiving flight-related data in real time and detecting anomalies using the machine learning model, means for generating countermeasures for the detected anomalies, and means for transmitting the generated anomaly notification and countermeasures to the operation terminal, thereby enabling real-time anomaly detection and rapid response, thereby improving the safety and efficiency of airline operations.

[0659] A "database" is a system designed for the purpose of organizing and storing information, and centrally manages past operational data, weather information, accident and incident history, and more.

[0660] A "machine learning model" is an algorithm that learns patterns based on large amounts of data and makes predictions and detects anomalies in new data.

[0661] "Flight-related data" refers to all information related to aircraft operations, including real-time flight status, weather information, and surveillance camera footage.

[0662] "Anomaly detection" is the process of identifying abnormal situations that deviate from normal operating conditions, allowing for prompt countermeasures to be taken.

[0663] "Countermeasures" refer to actions or measures taken in response to detected abnormalities, with the aim of solving problems or avoiding risks.

[0664] An "operation terminal" is an electronic device used by air traffic controllers and pilots, which notifies users of abnormalities, displays countermeasures, and receives user input.

[0665] A "communication device" is hardware and software for transmitting and receiving data, and is used to exchange real-time data between a server and a terminal.

[0666] "Data Processing Device" means a collection of hardware and software used to process and analyze collected data for anomaly detection and countermeasure generation.

[0667] This invention is a system for improving the safety and efficiency of airline operations, and aims to reduce human error by using machine learning. This system is built through collaboration between servers, terminals, and users, and enables real-time anomaly detection and response.

[0668] server

[0669] The server collects and processes large amounts of operational data, trains machine learning models, and receives and analyzes real-time data. Specifically, it uses the following hardware and software:

[0670] Hardware: High-performance server

[0671] Software: MySQL (database management), TensorFlow (building and training machine learning models), Kafka (receiving and processing real-time data)

[0672] The server collects past operation data from a MySQL database and uses the collected data to train a machine learning model using TensorFlow. The operation data received in real time is processed using Kafka to detect anomalies. If an anomaly is detected, the details are analyzed and a countermeasure is generated. This is all done automatically, and the generated anomaly notification and countermeasure are sent to the terminal.

[0673] For example, to train a forecasting model for severe weather, the server uses TensorFlow to learn how weather conditions affect flights based on 10 years of weather data. If a storm is approaching, the server immediately determines whether a particular flight is safe and notifies the device of the result.

[0674] Terminal

[0675] The terminal visually displays the data sent from the server and supports the user. The following hardware and software are used:

[0676] Hardware: High-resolution display, speakers

[0677] Software: HTML / CSS (dashboard display), JavaScript (asynchronous communication)

[0678] The terminal immediately displays the abnormality notification and countermeasures sent from the server and gives a voice notification. Information entered by the user and the judgment results are sent to the server in real time using JavaScript. The terminal also records operation logs, sends them to the server and stores them in a database.

[0679] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the surveillance video transmitted in real time and notifies the user of runway intrusion warnings visually and audibly. After the user performs a confirmation action, the information is sent to the server.

[0680] User

[0681] The user can make a decision based on the information displayed on the device and take the necessary action. Specific examples of operations are as follows:

[0682] The user checks the abnormality notification displayed on the terminal and decides on the appropriate countermeasure depending on the situation. The user inputs the countermeasure into the terminal and executes it. If necessary, the user can also input additional information or comments and send them to the server in real time.

[0683] For example, if a user sees information about an approaching storm, they can decide whether to delay, cancel, or reroute their flight and input their response plan into their device. The information is then sent to the server and shared with relevant parties immediately. If a false runway intrusion alarm is received, the user can quickly take action and input the results into their device and send them to the server.

[0684] Prompt Sentence Examples

[0685] "Learn how storms affect your flight using weather data from the past decade."

[0686] "Please create a system that provides visual and audio warnings for runway intrusions."

[0687] "Explain how flights will be delayed, canceled, or rerouted based on approaching storms."

[0688] This system significantly improves the safety and efficiency of air travel by linking servers, terminals, and users to detect and respond to abnormalities in real time.

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1:

[0691] The server collects historical flight data from the database. Specifically, it connects to the MySQL database and executes the SQL query "SELECT FROM flight_data WHERE date BETWEEN '2010-01-01' AND '2020-12-31';". The input is the SQL query, and the output is data on past flight records, weather information, and accidents and incidents. This output data is used as training data to build an anomaly detection model.

[0692] Step 2:

[0693] The server trains a machine learning model based on the collected data. Data preprocessing involves filling in missing values ​​and normalizing the data. Specifically, the model is trained using TensorFlow using "model.fit(training_data, labels, epochs=50)". The input is the preprocessed data, and the output is a trained anomaly detection model.

[0694] Step 3:

[0695] The server receives flight-related data in real time. Specifically, it uses Kafka to receive data via "consumer.poll(100)" and processes it as real-time data. The input is real-time flight status, weather information, and surveillance camera footage, and the output is packets containing this data.

[0696] Step 4:

[0697] The server inputs real-time data into a machine learning model for analysis. Specifically, it inputs the received data into "model.predict(new_flight_data)" to detect anomalies. The input is real-time data, and the output is the prediction result (presence or absence of anomalies).

[0698] Step 5:

[0699] When an anomaly is detected, the server analyzes the details and generates countermeasures. Specifically, it runs "analyze_anomaly(predictions)" to analyze the details of the anomaly and generate countermeasures. The input is the prediction result, and the output is an anomaly notification including countermeasures.

[0700] Step 6:

[0701] The server sends the generated abnormality notification and countermeasures to the operation terminal. Specifically, it uses Kafka to send notifications as follows: "producer.send('alerts', alert_message)". The input is the abnormality notification and countermeasures, and the output is a notification message to the operation terminal.

[0702] Step 7:

[0703] The terminal displays the data sent from the server on the screen. Specifically, a warning message is displayed on a dashboard created with HTML / CSS using "document.getElementById('alert').innerHTML = alert_message;". The input is the notification message from the server, and the output is the abnormality notification and countermeasures displayed on the terminal screen.

[0704] Step 8:

[0705] The device receives additional information and decision results entered by the user and sends them to the server via asynchronous communication. Specifically, it uses the JavaScript "fetch" API, using "fetch(' / submit-decisions', { method: 'POST', body: JSON.stringify(user_decision)})". The input is the user's decision result, and the output is the data to be sent to the server.

[0706] Step 9:

[0707] The terminal notifies the user of warnings and instructions from the server visually and audibly. Specifically, it uses the browser's notification API to display a warning with "new Notification('Warning', { body: 'An abnormality has been detected on the runway.'})". The input is the warning message from the server, and the output is the notification to the user.

[0708] Step 10:

[0709] The user makes a decision based on the information displayed on the terminal and takes the necessary action. Specifically, the user checks the warning message and inputs the action, such as flight delay, cancellation, or rerouting, into the terminal. The input is the information displayed on the terminal, and the output is the action taken and its results.

[0710] Step 11:

[0711] User operations and the details of the countermeasures taken are recorded on the terminal, and the data is sent to the server and stored in a database. Specifically, the operation log is collected in JSON format and sent to the server using "fetch(' / log-actions', { method: 'POST', body: JSON.stringify(action_log)})". The input is the user operation log, and the output is the recorded data stored in the database.

[0712] (Application example 1)

[0713] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] Conventional autonomous vehicle systems are inadequate in their real-time response to changes in traffic and weather conditions, making it difficult to detect abnormalities and generate appropriate countermeasures. This raises concerns about traffic accidents and reduced safety. Furthermore, they lack the ability to quickly incorporate user feedback and improve system accuracy. There is a need for a system that can solve these issues and improve the safety and efficiency of autonomous vehicles.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0716] In this invention, the server includes: means for collecting past operation data from a database; means for training an AI model for anomaly detection based on the collected data; means for receiving operation data in real time and detecting anomalies using the AI ​​model; means for generating countermeasures for the detected anomalies; means for transmitting the generated anomaly notification and countermeasures to a user terminal; means for collecting real-time sensor data and weather information from the autonomous vehicle; means for analyzing the collected data using the AI ​​model to detect abnormal conditions; means for displaying the anomaly notification and countermeasures visually and audibly on the user terminal; means for receiving user input and providing real-time feedback to the system; means for recording user operations and anomaly response details and saving the records in a database; and means for displaying the saved record data in a format that can be reviewed later. This makes it possible to monitor changes in traffic and weather conditions in real time while the autonomous vehicle is operating, quickly detect abnormal conditions, and automatically generate appropriate countermeasures, thereby improving safety and efficiency.

[0717] A "database" is a storage device that accumulates and stores past operation data, weather information, traffic accident data, and the like.

[0718] "Operation data" refers to all information that indicates the vehicle's operating status, such as various sensor data, camera footage, location information, and speed information from an autonomous vehicle.

[0719] An "AI model" is an algorithm trained to detect and predict anomalies based on collected data.

[0720] "Sensor data" refers to real-time operational information acquired by various sensors (e.g., cameras, lidar, radar) installed in autonomous vehicles.

[0721] "Weather information" refers to information about weather conditions during operation, such as wind speed, rainfall, and temperature.

[0722] An "abnormal state" is an unusual situation detected during operation (e.g., sudden deceleration, lane departure, approaching obstacle, etc.).

[0723] "Countermeasures" are specific actions or measures that should be taken in response to a detected abnormal condition.

[0724] A "user terminal" is a device (for example, an in-car display or a smartphone) that displays information sent from a server and receives user input.

[0725] "Feedback" is real-time input, such as additional information or decisions provided by the user, that is used to improve the system.

[0726] "Voice notification" is a means of conveying warnings and instructions to the user by voice.

[0727] A "visual display" is a means of displaying information using text and graphics on a user terminal display.

[0728] This invention is a system that improves the operational safety and efficiency of autonomous vehicles. It uses AI to monitor changes in traffic and weather conditions in real time, quickly detects abnormal conditions, and automatically generates appropriate countermeasures.

[0729] System Configuration

[0730] The system includes the following major components:

[0731] server

[0732] The server has the following roles:

[0733] 1. Database (accumulation of past operation data):

[0734] The database will store past driving data, weather information, traffic accident data, etc. This data is collected from autonomous vehicles in operation.

[0735] 2. Training the AI ​​model:

[0736] The server uses the collected data to train an AI model, which learns patterns of abnormal conditions and enables real-time anomaly detection.

[0737] 3. Real-time data reception and analysis:

[0738] The server receives sensor data and weather information sent from the autonomous vehicle in real time and analyzes it using an AI model.

[0739] 4. Detect anomalies and generate countermeasures:

[0740] If an anomaly is detected, the details are analyzed and countermeasures are generated, providing specific measures to keep the autonomous vehicle operating safely.

[0741] 5. Sending abnormality notification and countermeasures to user devices:

[0742] A notification of the detected abnormality and the generated countermeasure are sent to a user terminal inside the vehicle.

[0743] User devices (displays and smartphones inside autonomous vehicles)

[0744] The user terminal provides the following functions:

[0745] 1. Data display:

[0746] Displays abnormal notifications and countermeasures sent from the server visually and audibly.

[0747] 2. Receiving user input:

[0748] Receive feedback and additional information from the user and send that information to the server.

[0749] 3. Warning notice:

[0750] If an abnormality is detected, the user will be notified with a visual and audio warning.

[0751] 4. Data recording and archiving:

[0752] The user's operations and responses are recorded and the data is saved in a database.

[0753] User

[0754] The user roles are as follows:

[0755] 1. Check the data and take action:

[0756] Check for abnormality notifications and recommended actions, and take action as necessary.

[0757] 2. Providing Feedback:

[0758] Provide real-time feedback to the system to improve data accuracy.

[0759] Specific examples

[0760] For example, if an autonomous vehicle detects sudden deceleration while traveling on a highway, the system operates as follows:

[0761] 1. The server analyzes sudden deceleration data on the highway using an AI model and detects it as an abnormal condition.

[0762] 2. The server generates a countermeasure for the detected abnormal condition, such as "activate the automatic brakes to prevent deceleration."

[0763] 3. The generated abnormality notification and countermeasures are sent to the user terminal.

[0764] 4. The user device visually displays the abnormal condition and countermeasures on the in-car display and also notifies the driver audibly.

[0765] 5. The user acknowledges the notification and provides appropriate feedback to the system (e.g., "Got it, brake on").

[0766] 6. User actions are recorded and saved in a format that can be reviewed at a later date.

[0767] Example prompt sentence:

[0768] For example, if an AI model detects a sudden deceleration on a highway and generates a countermeasure, it might generate a prompt like this:

[0769] Abnormal condition: sudden deceleration

[0770] Location information: Expressway X section

[0771] Recommended action: Activate the automatic brakes to prevent slowing down.

[0772] Notification content: Sudden deceleration detected on X section of highway. Activate automatic brakes to prevent deceleration.

[0773] This will increase the safety and efficiency of self-driving vehicles.

[0774] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0775] Step 1:

[0776] The server collects historical driving data, weather information, traffic accident data, etc. from a database. The input is the database, and the output is raw data used to train the AI ​​model. Specifically, it uses SQL queries to retrieve the required data and saves it in CSV or JSON format.

[0777] Step 2:

[0778] The server trains an AI model based on the collected data. The input is the raw data collected in step 1, and the output is the trained AI model. Specifically, an anomaly detection model is built and trained using Python libraries such as Scikit-learn and TensorFlow.

[0779] Step 3:

[0780] The server receives sensor data and weather information sent from autonomous vehicles in real time. The input is real-time data, and the output is a dataset to be input into the AI ​​model. Specifically, the server receives real-time data via WebSocket or HTTP, preprocesses it, and formats it into a form that can be provided to the AI ​​model.

[0781] Step 4:

[0782] The server inputs the received real-time data into the AI ​​model for analysis. The input is a formatted data set, and the output is the analysis result (presence or absence of anomalies). Specifically, the predict method of the AI ​​model is called to detect abnormalities.

[0783] Step 5:

[0784] When the server detects an anomaly, it analyzes the details and generates a countermeasure. The input is the analysis result, and the output is a specific countermeasure and an anomaly notification message. Specifically, the generated anomaly notification and countermeasure are converted into JSON format and sent to the user's device.

[0785] Step 6:

[0786] The terminal displays the abnormality notification and countermeasures sent from the server visually and audibly. The input is the message from the server, and the output is the notification to the user. Specifically, the abnormality notification and countermeasures are displayed on the screen, and audio guidance is provided through the speaker.

[0787] Step 7:

[0788] The user checks the notified abnormal condition and takes appropriate action. The input is the notification from the terminal, and the output is the action taken and its results. Specifically, the user inputs feedback such as "Got it, apply brakes" into the terminal, which actually applies the brakes.

[0789] Step 8:

[0790] The device records the user's operations and the countermeasures taken, and sends the records to the server. The input is the user's feedback and operations, and the output is the recorded data. Specifically, the operations are saved in a log file and periodically uploaded to the server.

[0791] Step 9:

[0792] The server stores the recorded data in a database and displays it in a format that can be reviewed later. The input is the recorded data, and the output is the display items that can be reviewed by the user. Specifically, it inserts data into a database and visualizes the data in a web application or dashboard.

[0793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0794] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[0795] System configuration

[0796] The system includes the following major components:

[0797] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[0798] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[0799] 3. Server: Collects, processes and analyzes data.

[0800] 4. User terminal: An electronic device used by controllers and pilots to receive monitoring results and abnormality notifications.

[0801] 5. Emotion Engine: An engine that recognizes user emotions in real time and analyzes the data.

[0802] Program processing

[0803] server

[0804] 1. Data Collection:

[0805] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents, as well as user sentiment data.

[0806] 2. Training the AI ​​model:

[0807] The server trains an AI model based on the collected data. The AI ​​model learns patterns in the data to detect and predict anomalies. In particular, by including user emotional data, it becomes possible to provide emotion-based countermeasures for anomalies.

[0808] 3. Real-time data reception and analysis:

[0809] The server receives real-time operational data (flight information, aircraft status, weather information, etc.), provides this data to the AI ​​model, and analyzes the current operational status. The analysis results are used to monitor whether there are any operational anomalies.

[0810] 4. Detect anomalies and generate countermeasures:

[0811] The server uses an AI model to detect anomalies and identify their details (location, scope of impact, cause, etc.). It then generates countermeasures for the detected anomalies. Based on information from the emotion engine, it adjusts the priority of countermeasures according to the user's emotional state.

[0812] 5. Notification of abnormalities and response measures:

[0813] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[0814] Terminal

[0815] 1. Data display:

[0816] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[0817] 2. Warning Notice:

[0818] The terminal notifies the user of warnings and instructions from the server audibly and visually, records whether the user acknowledges the warning, and sends that information to the server.

[0819] 3. Receiving user input:

[0820] The terminal receives additional information and judgment results input by the user and transmits the data to the server in real time.

[0821] 4. Collecting Emotional Data:

[0822] The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[0823] 5. Data recording and archiving:

[0824] The device automatically records the user's operations and the countermeasures taken, and sends the recorded data to a server where it is stored in a database.

[0825] User

[0826] 1. Data review and situation assessment:

[0827] The user checks the information displayed on the device (such as abnormality notifications and countermeasures). Based on the displayed information, the user decides on the optimal countermeasure taking into account the actual operating conditions. The user can also refer to their own emotional data collected by the emotion engine.

[0828] 2. Enter and execute instructions:

[0829] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction. The details and results of the executed countermeasure are also input into the terminal, and the data is sent to the server.

[0830] 3. Warning confirmation and response:

[0831] The user checks the warnings and instructions sent from the device and takes the necessary action promptly. The device records the action and the situation, and sends that information to the server.

[0832] 4. Regular review and updates:

[0833] Users regularly check the monitoring data and warning information displayed on their devices, and take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0834] Specific examples

[0835] Server operation example

[0836] For example, when monitoring for abnormalities caused by bad weather, the server collects and learns from past bad weather data and analyzes it using an AI model. In addition, it also collects and learns data on the user's emotional state during past bad weather. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the results to the user's device. At this point, it also presents countermeasures prioritized according to the user's emotional state.

[0837] Example of terminal operation

[0838] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly, and the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[0839] Examples of user responses

[0840] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can refer to the analysis results of the emotion engine to select the most appropriate response. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, a user who receives an alert about a false intrusion on the runway can quickly implement a response and enter the results into the device. The emotion engine will evaluate the user's emotional state, and the results will also be used to improve the service.

[0841] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[0842] The processing flow will be explained below.

[0843] server

[0844] Step 1: Data collection

[0845] 1. The server collects past flight data, weather information, accident and incident history, and past emotion data from a database related to flight operations.

[0846] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model and emotion engine.

[0847] Step 2: Training the AI ​​model and emotion engine

[0848] 1. The server feeds the prepared dataset to the AI ​​model and emotion engine to train it for anomaly detection and emotion recognition.

[0849] 2. The server evaluates the trained AI model and emotion engine, and checks and adjusts its accuracy and processing power.

[0850] Step 3: Receive and analyze real-time data

[0851] 1. The server receives real-time operational data (flight information, aircraft status, weather information, etc.) and user emotion data.

[0852] 2. The server provides the received real-time data to the AI ​​model and emotion engine to analyze the current driving situation and the user's emotional state.

[0853] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation and monitors the user's emotional state.

[0854] Step 4: Detect anomalies and generate countermeasures

[0855] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[0856] 2. The server uses an emotion engine to evaluate the user's emotional state and reflects this in the anomaly detection results.

[0857] 3. The server generates countermeasures for the detected anomalies and adjusts the priority of the countermeasures according to the user's emotional state.

[0858] Step 5: Sending notification of abnormalities and countermeasures

[0859] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[0860] Terminal

[0861] Step 1: Data display

[0862] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[0863] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[0864] Step 2: Notification of warnings

[0865] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[0866] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[0867] Step 3: Receiving User Input

[0868] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[0869] Step 4: Collecting emotion data

[0870] 1. The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[0871] Step 5: Record and archive data

[0872] 1. The device automatically records user operations and the countermeasures taken.

[0873] 2. The terminal periodically sends the recorded data to the server and stores it in a database.

[0874] User

[0875] Step 1: Check the data and assess the situation

[0876] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[0877] 2. Based on the displayed information, the user decides on the best course of action to take, taking into account the actual driving situation and their own emotional state.

[0878] Step 2: Enter and execute instructions

[0879] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[0880] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[0881] Step 3: Check for warnings and take action

[0882] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[0883] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[0884] Step 4: Regularly check and update

[0885] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[0886] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[0887] Specific examples

[0888] Server operation example

[0889] For example, when monitoring for abnormalities caused by bad weather, the server collects past bad weather data and analyzes it using an AI model. It also collects data on the user's emotional state during bad weather and analyzes it using an emotion engine. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of a specific flight and sends the results to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[0890] Example of terminal operation

[0891] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly. At this time, the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, along with the analyzed emotion data.

[0892] Examples of user responses

[0893] For example, a user who has received information about an approaching storm may decide to delay or cancel their flight, or to choose an alternative route. In this case, the user may refer to the analysis results of the emotion engine to select the optimal response. The selected response is then entered into the device, and the information is sent to the server and shared with relevant parties immediately. Furthermore, the user's emotional state is also recorded and analyzed, and this information is used to improve the service.

[0894] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[0895] Example 2

[0896] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0897] Conventional systems for detecting anomalies and providing countermeasures in aviation operations are prone to human error and place a heavy psychological burden on users. In particular, in emergency situations, the user's emotional state is not properly taken into account, which can result in the selection of the most appropriate countermeasure. This has led to a demand for improved safety and efficiency.

[0898] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0899] In this invention, the server includes means for collecting past operation data and emotion data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures based on the detected anomalies and the user's emotional state, and means for transmitting the generated anomaly notification and countermeasures to the user terminal. This makes it possible to provide anomaly countermeasures that take the user's emotional state into consideration, thereby achieving improved safety and efficiency.

[0900] The "database" is an information collection point for storing and managing various data required by the system, such as past operation data and emotion data.

[0901] "Emotion data" is information that indicates the user's psychological state, and is data extracted from facial expressions, tone of voice, behavior, and the like.

[0902] An "AI model" is an algorithm that is trained based on collected data to detect anomalies and make predictions.

[0903] "Server" means a central control unit that collects, processes, and analyzes data and manages communications with user terminals.

[0904] A "user terminal" is an electronic device used by air traffic controllers and pilots to receive information and implement countermeasures.

[0905] "Anomaly detection" is the process of using AI models to analyze operational data and identify abnormal conditions or malfunctions.

[0906] "Countermeasures" refer to specific actions or instructions to be taken in response to detected abnormalities.

[0907] The "emotion engine" is software that analyzes the user's facial expressions and tone of voice, and collects and evaluates emotional data in real time.

[0908] An "abnormality notification" is a message or notification that notifies a user terminal of a detected abnormality and a countermeasure.

[0909] "Users" are human operators such as controllers and pilots who operate the system, review information, and implement countermeasures.

[0910] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[0911] The system includes the following major components:

[0912] 1. Database

[0913] It stores past operation data, weather information, accident and incident history, user emotion data, etc. To do this, it uses a commonly used relational database (e.g., MySQL).

[0914] 2. AI Model

[0915] This is an algorithm trained on database information to perform anomaly detection and prediction. It uses the Python TensorFlow library. In particular, by including user emotional data, it is possible to provide emotion-based countermeasures for anomalies.

[0916] 3. Server

[0917] It collects, processes, and analyzes data. It receives operational data (flight information, aircraft status, weather information, etc.) in real time and provides it to an AI model to analyze operational status. The server receives real-time data using WebSocket technology and sends the analysis results to the user's device.

[0918] 4. User Device

[0919] This is an electronic device used by air traffic controllers and pilots to receive monitoring results and abnormality notifications. The terminal displays information sent from the server in real time and issues audio and visual warnings. It also sends additional information and judgment results entered by the user to the server in real time, and uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data.

[0920] 5. Emotion Engine

[0921] This is an engine that recognizes user emotions in real time and analyzes the data. The emotion engine uses a camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. The collected data is sent to a server and used to generate countermeasures against abnormalities.

[0922] Specific examples

[0923] Server operation example

[0924] For example, when a server monitors for abnormalities caused by bad weather, it collects and learns from past bad weather data and analyzes it using an AI model. It also collects and learns data on the user's emotional state during past bad weather. If it analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the result to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[0925] Example of terminal operation

[0926] For example, if a device monitors a runway using a surveillance camera, it will display the video in real time and notify the user visually and audibly of any runway intrusion warnings from the server. At this time, the emotion engine analyzes the user's facial expressions and tone of voice, and when the user acknowledges the warning, that information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[0927] Examples of user responses

[0928] For example, if a user sees information about an approaching storm, they can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can select the optimal response, taking into account the analysis results of the emotion engine. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, if a user receives an alert about a false runway intrusion, they can quickly implement a response and enter the results into the device. The emotion engine will then evaluate the user's emotional state, and the results will also be used to improve the service.

[0929] Prompt Sentence Examples

[0930] "Please explain how the server collects historical severe weather data and operational data to train the AI ​​model. Also, please describe the process by which the weather information is analyzed in real time and anomaly response measures are generated based on the user's emotional state."

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Program processing flow

[0933] server

[0934] Step 1:

[0935] Data collection

[0936] The server connects to the flight database to retrieve flight records, weather information, past accident and incident data, and user sentiment data. These data are retrieved using a MySQL database.

[0937] Input: Database connection information

[0938] Processing: Extract the required data from the database using SQL queries.

[0939] Output: A set of operational and emotional data

[0940] Step 2:

[0941] Training an AI model

[0942] The server uses the Python TensorFlow library to train the AI ​​model, which learns patterns to detect anomalies based on collected operational and emotional data.

[0943] Input: A set of operational data and emotion data

[0944] Processing: After data preprocessing (normalization, feature extraction, etc.), training is performed using an AI model.

[0945] Output: A trained AI model

[0946] Step 3:

[0947] Receiving and analyzing real-time data

[0948] The server periodically receives real-time data (flight information, aircraft status, weather data, etc.) sent from the aircraft using WebSocket technology, and provides the received data to an AI model to analyze the current operating status.

[0949] Input: Real-time data

[0950] Processing: The received data is fed into an AI model for analysis to determine whether there are any anomalies.

[0951] Output: Current operating status and abnormality detection results

[0952] Step 4:

[0953] Detecting anomalies and generating countermeasures

[0954] The server uses an AI model to detect anomalies. If an anomaly is detected, it identifies the details (location of occurrence, scope of impact, cause, etc.), adjusts the priority of countermeasures based on data from the emotion engine, and then generates countermeasures.

[0955] Input: Anomaly detection results and emotion data

[0956] Action: Generate and prioritize countermeasures based on detected anomalies.

[0957] Output: Abnormal notification and countermeasures

[0958] Step 5:

[0959] Sending abnormality notifications and countermeasures

[0960] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[0961] Input: Abnormal notification and countermeasures

[0962] Processing: Convert the notification data into an appropriate format and send it to the user terminal.

[0963] Output: Abnormal notification to user terminal

[0964] Terminal

[0965] Step 1:

[0966] Data Display

[0967] The terminal displays the monitoring results and abnormality notifications sent from the server in real time, using an interface that uses a GUI library.

[0968] Input: Abnormal notification and countermeasures

[0969] Processing: Updates various display elements of the GUI.

[0970] Output: Displayed abnormality notification and countermeasures

[0971] Step 2:

[0972] Warning Notification

[0973] The terminal will notify the user of warnings and instructions from the server audibly and visually, including playing audio files and visual alerts.

[0974] Input: Error notification and warning information

[0975] Action: Plays an audio and displays a visual alert.

[0976] Output: Audio and visual warnings

[0977] Step 3:

[0978] Receiving User Input

[0979] The terminal receives input from the user and transmits the data to the server in real time, either via a touchscreen or keyboard input.

[0980] Input: User-entered data

[0981] Processing: Sends the received user input to the server.

[0982] Output: Send input data to the server

[0983] Step 4:

[0984] Collecting Emotional Data

[0985] The device uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data, using a camera and microphone.

[0986] Input: User's facial expression data and voice data

[0987] Processing: Emotion data is generated using facial expression analysis algorithms and voice analysis algorithms.

[0988] Output: Emotion data

[0989] Step 5:

[0990] Data recording and archiving

[0991] The device automatically records the user's actions and the countermeasures taken, and transmits the data to the server using local storage and network communication.

[0992] Input: User operation log and execution details

[0993] Processing: The log data is stored in a format and sent to the server.

[0994] Output: Send log data to the server

[0995] User

[0996] Step 1:

[0997] Data confirmation and situation assessment

[0998] The user checks the information displayed on the device, determines the best course of action based on the operational status, and takes appropriate action based on the displayed information.

[0999] Input: Abnormality notification and countermeasures displayed on the terminal

[1000] Processing: Analyze the displayed information and make decisions based on the situation.

[1001] Output: Decision results and action selection

[1002] Step 2:

[1003] Entering and executing instructions

[1004] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction.

[1005] Input: Workaround

[1006] Processing: Executes the input instructions and inputs the results to the terminal.

[1007] Output: Input data and execution results to the terminal

[1008] Step 3:

[1009] Checking and responding to warnings

[1010] The user checks the warnings and instructions from the terminal and takes appropriate action promptly.

[1011] Input: Warning notices and instructions

[1012] Action: Assess the situation and take the most appropriate action.

[1013] Output: Countermeasure execution results

[1014] Step 4:

[1015] Regular review and updates

[1016] Users regularly check the monitoring data and warning information displayed on their devices and take preventive measures or make advance preparations as necessary.

[1017] Input: Monitoring data and warning information

[1018] Processing: Regularly review the data and take any necessary action.

[1019] Output: Preventive measures taken and preparatory results

[1020] (Application example 2)

[1021] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1022] Conventional autonomous vehicle traffic management systems can detect anomalies and generate countermeasures in real time, but they cannot consider the emotional state of passengers and drivers. As a result, psychological burden increases and appropriate responses are delayed. Therefore, in order to improve safety and efficiency during operation, it is necessary to analyze the user's emotional state in real time and provide appropriate countermeasures.

[1023] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past operation data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures for the detected anomaly, means for transmitting the generated anomaly notification and countermeasures to the user terminal, means for analyzing the user's emotional state using an emotion engine, and means for adjusting the priority of the countermeasures based on the analyzed emotional state. This makes it possible to provide anomaly countermeasures according to the user's emotional state.

[1024] The "database" is an information management system for storing past operational data, weather information, accident and incident history, and user emotional data.

[1025] "Operation data" refers to information about the operating status of an autonomous vehicle, including, for example, flight records, weather information, and vehicle sensor data.

[1026] An "AI model" is an algorithm that is trained on information from a database to perform anomaly detection and prediction.

[1027] "Anomaly detection" is the process of using AI models to identify unusual behaviors or situations in operational data.

[1028] "Countermeasures" are specific measures or action plans for the detected abnormality, and are intended to ensure safety and improve operational efficiency.

[1029] A "user terminal" is a device used by controllers and drivers, and is an electronic device for receiving monitoring results and abnormality notifications.

[1030] The "emotion engine" is a system that analyzes the user's emotional state in real time and reflects that data in operational management.

[1031] "Priority" is an index that indicates the importance and urgency of implementing an abnormality response measure, and is adjusted based on the emotional state of the user.

[1032] An "abnormality notification" is a message for transmitting information about a detected abnormality to a user terminal.

[1033] "Analysis" is the process of identifying and understanding situations and patterns based on collected data.

[1034] The present invention combines an emotion engine with a safe driving management system for autonomous vehicles, making it possible to provide countermeasures against abnormalities that take into account the emotional state of the user. A specific embodiment of the system for implementing the present invention will be described below.

[1035] System configuration

[1036] The system includes the following major components:

[1037] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[1038] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1039] 3. Server: Collects, processes and analyzes data.

[1040] 4. User terminal: An electronic device used by the driver or passenger to receive monitoring results and abnormality notifications.

[1041] 5. Emotion engine: Analyzes user emotions in real time and reflects that data in operation management.

[1042] Program processing

[1043] server

[1044] The server collects historical driving data (e.g., speed, location, and surrounding obstacle information) from the driving database. This data includes vehicle sensor data, real-time weather data, traffic condition data, and user emotion data. Based on the collected data, the server trains an AI model to detect and predict anomalies. Furthermore, the server receives real-time driving data, detects anomalies using the AI ​​model, and generates countermeasures. The server adjusts the priority of countermeasures based on the emotion data analyzed by the emotion engine.

[1045] User terminal

[1046] The user device receives the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server and notifies the user visually and audibly. The device also uses a camera and smart glasses to analyze the user's facial expressions and tone of voice to collect emotional data. This allows the user's emotional state to be sent to the server in real time and used to adjust the priority of countermeasures to anomalies.

[1047] User

[1048] The user checks the information displayed on the device (such as abnormality notifications and countermeasures) and selects and executes the most appropriate countermeasure for the situation. In doing so, the user also takes into account their own emotional data analyzed by the emotion engine. For example, after checking information about an approaching storm, the user can decide whether to delay or cancel a flight or select an alternative route. The countermeasures taken are entered into the device and recorded and analyzed on the server.

[1049] Specific examples

[1050] For example, when an autonomous vehicle detects abnormal data during bad weather, the system predicts the abnormality based on past weather data and the user's emotional state at the time. If passenger anxiety increases due to the emotion engine, the system can quickly provide high-priority countermeasures such as delays or route changes. After the user confirms and acts on the warning, the information is sent to a server and stored in a database.

[1051] Example prompts to input to the generative AI model

[1052] Get up-to-date traffic data, calculate safe and efficient routes based on weather data and vehicle sensor data, and take into account the emotional state (e.g., stress, relief) of the passengers and driver in the car.

[1053] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1054] Step 1:

[1055] The server collects historical driving data from a database. This data includes vehicle sensor data (speed, location, obstacle information, etc.), weather data, traffic condition data, and user emotion data. This creates a dataset that provides a comprehensive understanding of driving conditions. The collected data is used to train AI models for anomaly detection and prediction.

[1056] Input: Historical driving data, sensor data, weather data, traffic data, user emotion data

[1057] Output: Collected dataset

[1058] Step 2:

[1059] The server trains the AI ​​model based on the collected data. This training process learns patterns in the data and builds algorithms capable of detecting and predicting anomalies. In particular, by including user emotional data, it is possible to generate countermeasures to anomalies that take into account the user's emotional state.

[1060] Input: Collected dataset

[1061] Output: A trained AI model

[1062] Step 3:

[1063] The server receives real-time operational data, including the vehicle's current sensor data, real-time weather information, and traffic data, which is then fed into an AI model to analyze the current operational situation.

[1064] Input: Real-time operation data, sensor data, weather information, traffic condition data

[1065] Output: Parsed operational status data

[1066] Step 4:

[1067] The server uses an AI model to detect anomalies. It then identifies the details of the detected anomaly (location, scope of impact, cause, etc.) and generates countermeasures. It then adjusts the priority of countermeasures based on the user's emotional data collected by the emotion engine.

[1068] Input: Analyzed driving status data, real-time emotion data

[1069] Output: Coordinated response measures, anomaly notification

[1070] Step 5:

[1071] The server sends the generated abnormality notification and countermeasures to the user terminal. The user terminal receives this and notifies the user visually and audibly. The user checks the information displayed on the terminal and selects and executes the appropriate countermeasures.

[1072] Input: Coordinated action, anomaly notification

[1073] Output: User's action selection information

[1074] Step 6:

[1075] The user device uses a camera and smart glasses to analyze the user's facial expressions and tone of voice, collecting emotional data in real time. This emotional data is sent to a server and used to prioritize countermeasures against abnormalities.

[1076] Input: Real-time video and audio data of the user

[1077] Output: Real-time emotion data

[1078] Step 7:

[1079] The user enters the details and results of the selected action into the device and sends the data to the server, which stores this information in a database and uses it for future analysis and service improvement.

[1080] Input: User-entered details of countermeasures, execution results

[1081] Output: Details of saved countermeasures, execution result data

[1082] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1083] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1084] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1085] [Third embodiment]

[1086] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1087] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1088] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1089] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1090] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1091] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1092] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1093] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1094] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1095] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1096] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1097] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1098] This invention relates to an air navigation support system, which utilizes AI to support monitoring, guidance, and decision-making in air operations, with the aim of reducing human error and improving the safety and efficiency of air operations.

[1099] System configuration

[1100] The system includes the following major components:

[1101] 1. Database: Stores past operation data, weather information, accident and incident history, etc.

[1102] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1103] 3. Server: Collects, processes and analyzes data.

[1104] 4. User terminal: Electronic equipment used by controllers and pilots to receive monitoring results and abnormality notifications.

[1105] Program processing

[1106] server

[1107] 1. Data Collection:

[1108] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents.

[1109] 2. Training the AI ​​model:

[1110] The server trains an AI model based on the collected data, which learns patterns in the data to detect and predict anomalies.

[1111] 3. Real-time data reception and analysis:

[1112] The server receives real-time operational data, such as current weather information, flight status, and surveillance camera footage, and provides this data to the AI ​​model, which monitors and analyzes it for any abnormalities.

[1113] 4. Anomaly detection and notification:

[1114] When an anomaly is detected by the AI ​​model, the server analyzes the details and generates a countermeasure, then sends the generated anomaly notification and countermeasure to the user's device.

[1115] Terminal

[1116] 1. Data display:

[1117] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[1118] 2. Receiving user input:

[1119] The terminal receives additional information and judgment results input by the user and transmits them to the server, thereby updating the information in real time.

[1120] 3. Warning Notice:

[1121] The terminal notifies the user of warnings and instructions from the server by voice or visually, and records whether the user acknowledges the warning or not, and sends the record to the server.

[1122] 4. Data recording and archiving:

[1123] The device automatically records the user's actions and the countermeasures taken, sends the data to a server, stores it in a database, and displays it in a format that can be reviewed later.

[1124] User

[1125] 1. Data verification and judgment:

[1126] The user checks the information displayed on the device (for example, an abnormality notification or recommended countermeasures). Based on the displayed information, the user decides on the most appropriate countermeasure in light of the actual situation.

[1127] 2. Enter and execute instructions:

[1128] The user inputs the response they have decided on into the terminal and executes the instructions. By inputting additional information and comments and sending them to the server, more precise responses can be made.

[1129] 3. Warning confirmation and response:

[1130] The user checks the warnings and instructions (for example, an anomaly detection notification) from the terminal and takes the necessary action. The results of the action and the details of the action are entered into the terminal and sent to the server.

[1131] 4. Regular checks:

[1132] Users regularly check the monitoring data and warning information displayed on their devices. Even if no abnormalities are detected, they can take appropriate precautions and check for countermeasures according to the situation.

[1133] Specific examples

[1134] Server operation example

[1135] For example, when monitoring for bad weather, the server collects data on bad weather over the past 10 years and uses an AI model to learn what weather conditions will affect flights. When the server analyzes weather information in real time and detects an approaching storm, it determines whether a particular flight is safe and sends the result to the user's device.

[1136] Example of terminal operation

[1137] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the images transmitted in real time and notifies the user visually and audibly of runway intrusion warnings from the server. After the user performs a confirmation action, the information is sent to the server.

[1138] Examples of user responses

[1139] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight or choose an alternative route, and input their response into their device. The information is then sent to the server and shared immediately with relevant parties. Similarly, a user who receives a false runway intrusion alert can quickly implement a response, input the results into their device, and send them to the server.

[1140] As described above, the system of the present invention utilizes AI to support rapid and appropriate responses while minimizing human error in order to improve the safety and efficiency of operations.

[1141] The processing flow will be explained below.

[1142] server

[1143] Step 1: Data collection

[1144] 1. The server collects historical flight data from a database of flight operations.

[1145] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model.

[1146] Step 2: Training the AI ​​model

[1147] 1. The server feeds the prepared dataset to the AI ​​model and trains the model for anomaly detection and prediction.

[1148] 2. The server evaluates the trained AI model and checks and adjusts its accuracy and processing power.

[1149] Step 3: Receive and analyze real-time data

[1150] 1. The server receives operational data (flight information, aircraft status, weather information, etc.) in real time.

[1151] 2. The server provides the received real-time data to the AI ​​model and analyzes the current operating conditions.

[1152] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation.

[1153] Step 4: Detect anomalies and generate countermeasures

[1154] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[1155] 2. The server generates countermeasures for the detected anomalies.

[1156] Step 5: Sending notification and countermeasures

[1157] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[1158] Terminal

[1159] Step 1: Data display

[1160] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[1161] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[1162] Step 2: Notification of warnings

[1163] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[1164] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[1165] Step 3: Receiving input from the user

[1166] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[1167] Step 4: Record and archive data

[1168] 1. The device automatically records user operations and the countermeasures taken.

[1169] 2. The device sends the recorded data to the server and stores it in a database.

[1170] User

[1171] Step 1: Check the data and assess the situation

[1172] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[1173] 2. Based on the displayed information, the user decides on the best course of action to take in light of the actual operating conditions.

[1174] Step 2: Enter and execute instructions

[1175] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[1176] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[1177] Step 3: Check for warnings and take action

[1178] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[1179] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[1180] Step 4: Regularly check and update

[1181] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[1182] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1183] Example 1

[1184] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1185] To improve the safety and efficiency of air travel, there is a need for real-time anomaly detection and rapid response while minimizing human error. However, conventional systems lack the accuracy to process large amounts of operational data and detect anomalies, making it difficult to respond in real time. This can lead to reduced safety and operational delays, so a new system is needed to solve these issues.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1187] In this invention, the server includes means for collecting past flight data from a database, means for training a machine learning model for anomaly detection based on the collected data, means for receiving flight-related data in real time and detecting anomalies using the machine learning model, means for generating countermeasures for the detected anomalies, and means for transmitting the generated anomaly notification and countermeasures to the operation terminal, thereby enabling real-time anomaly detection and rapid response, thereby improving the safety and efficiency of airline operations.

[1188] A "database" is a system designed for the purpose of organizing and storing information, and centrally manages past operational data, weather information, accident and incident history, and more.

[1189] A "machine learning model" is an algorithm that learns patterns based on large amounts of data and makes predictions and detects anomalies in new data.

[1190] "Flight-related data" refers to all information related to aircraft operations, including real-time flight status, weather information, and surveillance camera footage.

[1191] "Anomaly detection" is the process of identifying abnormal situations that deviate from normal operating conditions, allowing for prompt countermeasures to be taken.

[1192] "Countermeasures" refer to actions or measures taken in response to detected abnormalities, with the aim of solving problems or avoiding risks.

[1193] An "operation terminal" is an electronic device used by air traffic controllers and pilots, which notifies users of abnormalities, displays countermeasures, and receives user input.

[1194] A "communication device" is hardware and software for transmitting and receiving data, and is used to exchange real-time data between a server and a terminal.

[1195] "Data Processing Device" means a collection of hardware and software used to process and analyze collected data for anomaly detection and countermeasure generation.

[1196] This invention is a system for improving the safety and efficiency of airline operations, and aims to reduce human error by using machine learning. This system is built through collaboration between servers, terminals, and users, and enables real-time anomaly detection and response.

[1197] server

[1198] The server collects and processes large amounts of operational data, trains machine learning models, and receives and analyzes real-time data. Specifically, it uses the following hardware and software:

[1199] Hardware: High-performance server

[1200] Software: MySQL (database management), TensorFlow (building and training machine learning models), Kafka (receiving and processing real-time data)

[1201] The server collects past operation data from a MySQL database and uses the collected data to train a machine learning model using TensorFlow. The operation data received in real time is processed using Kafka to detect anomalies. If an anomaly is detected, the details are analyzed and a countermeasure is generated. This is all done automatically, and the generated anomaly notification and countermeasure are sent to the terminal.

[1202] For example, to train a forecasting model for severe weather, the server uses TensorFlow to learn how weather conditions affect flights based on 10 years of weather data. If a storm is approaching, the server immediately determines whether a particular flight is safe and notifies the device of the result.

[1203] Terminal

[1204] The terminal visually displays the data sent from the server and supports the user. The following hardware and software are used:

[1205] Hardware: High-resolution display, speakers

[1206] Software: HTML / CSS (dashboard display), JavaScript (asynchronous communication)

[1207] The terminal immediately displays the abnormality notification and countermeasures sent from the server and gives a voice notification. Information entered by the user and the judgment results are sent to the server in real time using JavaScript. The terminal also records operation logs, sends them to the server and stores them in a database.

[1208] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the surveillance video transmitted in real time and notifies the user of runway intrusion warnings visually and audibly. After the user performs a confirmation action, the information is sent to the server.

[1209] User

[1210] The user can make a decision based on the information displayed on the device and take the necessary action. Specific examples of operations are as follows:

[1211] The user checks the abnormality notification displayed on the terminal and decides on the appropriate countermeasure depending on the situation. The user inputs the countermeasure into the terminal and executes it. If necessary, the user can also input additional information or comments and send them to the server in real time.

[1212] For example, if a user sees information about an approaching storm, they can decide whether to delay, cancel, or reroute their flight and input their response plan into their device. The information is then sent to the server and shared with relevant parties immediately. If a false runway intrusion alarm is received, the user can quickly take action and input the results into their device and send them to the server.

[1213] Prompt Sentence Examples

[1214] "Learn how storms affect your flight using weather data from the past decade."

[1215] "Please create a system that provides visual and audio warnings for runway intrusions."

[1216] "Explain how flights will be delayed, canceled, or rerouted based on approaching storms."

[1217] This system significantly improves the safety and efficiency of air travel by linking servers, terminals, and users to detect and respond to abnormalities in real time.

[1218] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1219] Step 1:

[1220] The server collects historical flight data from the database. Specifically, it connects to the MySQL database and executes the SQL query "SELECT FROM flight_data WHERE date BETWEEN '2010-01-01' AND '2020-12-31';". The input is the SQL query, and the output is data on past flight records, weather information, and accidents and incidents. This output data is used as training data to build an anomaly detection model.

[1221] Step 2:

[1222] The server trains a machine learning model based on the collected data. Data preprocessing involves filling in missing values ​​and normalizing the data. Specifically, the model is trained using TensorFlow using "model.fit(training_data, labels, epochs=50)". The input is the preprocessed data, and the output is a trained anomaly detection model.

[1223] Step 3:

[1224] The server receives flight-related data in real time. Specifically, it uses Kafka to receive data via "consumer.poll(100)" and processes it as real-time data. The input is real-time flight status, weather information, and surveillance camera footage, and the output is packets containing this data.

[1225] Step 4:

[1226] The server inputs real-time data into a machine learning model for analysis. Specifically, it inputs the received data into "model.predict(new_flight_data)" to detect anomalies. The input is real-time data, and the output is the prediction result (presence or absence of anomalies).

[1227] Step 5:

[1228] When an anomaly is detected, the server analyzes the details and generates countermeasures. Specifically, it runs "analyze_anomaly(predictions)" to analyze the details of the anomaly and generate countermeasures. The input is the prediction result, and the output is an anomaly notification including countermeasures.

[1229] Step 6:

[1230] The server sends the generated abnormality notification and countermeasures to the operation terminal. Specifically, it uses Kafka to send notifications as follows: "producer.send('alerts', alert_message)". The input is the abnormality notification and countermeasures, and the output is a notification message to the operation terminal.

[1231] Step 7:

[1232] The terminal displays the data sent from the server on the screen. Specifically, a warning message is displayed on a dashboard created with HTML / CSS using "document.getElementById('alert').innerHTML = alert_message;". The input is the notification message from the server, and the output is the abnormality notification and countermeasures displayed on the terminal screen.

[1233] Step 8:

[1234] The device receives additional information and decision results entered by the user and sends them to the server via asynchronous communication. Specifically, it uses the JavaScript "fetch" API, using "fetch(' / submit-decisions', { method: 'POST', body: JSON.stringify(user_decision)})". The input is the user's decision result, and the output is the data to be sent to the server.

[1235] Step 9:

[1236] The terminal notifies the user of warnings and instructions from the server visually and audibly. Specifically, it uses the browser's notification API to display a warning with "new Notification('Warning', { body: 'An abnormality has been detected on the runway.'})". The input is the warning message from the server, and the output is the notification to the user.

[1237] Step 10:

[1238] The user makes a decision based on the information displayed on the terminal and takes the necessary action. Specifically, the user checks the warning message and inputs the action, such as flight delay, cancellation, or rerouting, into the terminal. The input is the information displayed on the terminal, and the output is the action taken and its results.

[1239] Step 11:

[1240] User operations and the details of the countermeasures taken are recorded on the terminal, and the data is sent to the server and stored in a database. Specifically, the operation log is collected in JSON format and sent to the server using "fetch(' / log-actions', { method: 'POST', body: JSON.stringify(action_log)})". The input is the user operation log, and the output is the recorded data stored in the database.

[1241] (Application example 1)

[1242] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1243] Conventional autonomous vehicle systems are inadequate in their real-time response to changes in traffic and weather conditions, making it difficult to detect abnormalities and generate appropriate countermeasures. This raises concerns about traffic accidents and reduced safety. Furthermore, they lack the ability to quickly incorporate user feedback and improve system accuracy. There is a need for a system that can solve these issues and improve the safety and efficiency of autonomous vehicles.

[1244] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1245] In this invention, the server includes: means for collecting past operation data from a database; means for training an AI model for anomaly detection based on the collected data; means for receiving operation data in real time and detecting anomalies using the AI ​​model; means for generating countermeasures for the detected anomalies; means for transmitting the generated anomaly notification and countermeasures to a user terminal; means for collecting real-time sensor data and weather information from the autonomous vehicle; means for analyzing the collected data using the AI ​​model to detect abnormal conditions; means for displaying the anomaly notification and countermeasures visually and audibly on the user terminal; means for receiving user input and providing real-time feedback to the system; means for recording user operations and anomaly response details and saving the records in a database; and means for displaying the saved record data in a format that can be reviewed later. This makes it possible to monitor changes in traffic and weather conditions in real time while the autonomous vehicle is operating, quickly detect abnormal conditions, and automatically generate appropriate countermeasures, thereby improving safety and efficiency.

[1246] A "database" is a storage device that accumulates and stores past operation data, weather information, traffic accident data, and the like.

[1247] "Operation data" refers to all information that indicates the vehicle's operating status, such as various sensor data, camera footage, location information, and speed information from an autonomous vehicle.

[1248] An "AI model" is an algorithm trained to detect and predict anomalies based on collected data.

[1249] "Sensor data" refers to real-time operational information acquired by various sensors (e.g., cameras, lidar, radar) installed in autonomous vehicles.

[1250] "Weather information" refers to information about weather conditions during operation, such as wind speed, rainfall, and temperature.

[1251] An "abnormal state" is an unusual situation detected during operation (e.g., sudden deceleration, lane departure, approaching obstacle, etc.).

[1252] "Countermeasures" are specific actions or measures that should be taken in response to a detected abnormal condition.

[1253] A "user terminal" is a device (for example, an in-car display or a smartphone) that displays information sent from a server and receives user input.

[1254] "Feedback" is real-time input, such as additional information or decisions provided by the user, that is used to improve the system.

[1255] "Voice notification" is a means of conveying warnings and instructions to the user by voice.

[1256] A "visual display" is a means of displaying information using text and graphics on a user terminal display.

[1257] This invention is a system that improves the operational safety and efficiency of autonomous vehicles. It uses AI to monitor changes in traffic and weather conditions in real time, quickly detects abnormal conditions, and automatically generates appropriate countermeasures.

[1258] System Configuration

[1259] The system includes the following major components:

[1260] server

[1261] The server has the following roles:

[1262] 1. Database (accumulation of past operation data):

[1263] The database will store past driving data, weather information, traffic accident data, etc. This data is collected from autonomous vehicles in operation.

[1264] 2. Training the AI ​​model:

[1265] The server uses the collected data to train an AI model, which learns patterns of abnormal conditions and enables real-time anomaly detection.

[1266] 3. Real-time data reception and analysis:

[1267] The server receives sensor data and weather information sent from the autonomous vehicle in real time and analyzes it using an AI model.

[1268] 4. Detect anomalies and generate countermeasures:

[1269] If an anomaly is detected, the details are analyzed and countermeasures are generated, providing specific measures to keep the autonomous vehicle operating safely.

[1270] 5. Sending abnormality notification and countermeasures to user devices:

[1271] A notification of the detected abnormality and the generated countermeasure are sent to a user terminal inside the vehicle.

[1272] User devices (displays and smartphones inside autonomous vehicles)

[1273] The user terminal provides the following functions:

[1274] 1. Data display:

[1275] Displays abnormal notifications and countermeasures sent from the server visually and audibly.

[1276] 2. Receiving user input:

[1277] Receive feedback and additional information from the user and send that information to the server.

[1278] 3. Warning notice:

[1279] If an abnormality is detected, the user will be notified with a visual and audio warning.

[1280] 4. Data recording and archiving:

[1281] The user's operations and responses are recorded and the data is saved in a database.

[1282] User

[1283] The user roles are as follows:

[1284] 1. Check the data and take action:

[1285] Check for abnormality notifications and recommended actions, and take action as necessary.

[1286] 2. Providing Feedback:

[1287] Provide real-time feedback to the system to improve data accuracy.

[1288] Specific examples

[1289] For example, if an autonomous vehicle detects sudden deceleration while traveling on a highway, the system operates as follows:

[1290] 1. The server analyzes sudden deceleration data on the highway using an AI model and detects it as an abnormal condition.

[1291] 2. The server generates a countermeasure for the detected abnormal condition, such as "activate the automatic brakes to prevent deceleration."

[1292] 3. The generated abnormality notification and countermeasures are sent to the user terminal.

[1293] 4. The user device visually displays the abnormal condition and countermeasures on the in-car display and also notifies the driver audibly.

[1294] 5. The user acknowledges the notification and provides appropriate feedback to the system (e.g., "Got it, brake on").

[1295] 6. User actions are recorded and saved in a format that can be reviewed at a later date.

[1296] Example prompt sentence:

[1297] For example, if an AI model detects a sudden deceleration on a highway and generates a countermeasure, it might generate a prompt like this:

[1298] Abnormal condition: sudden deceleration

[1299] Location information: Expressway X section

[1300] Recommended action: Activate the automatic brakes to prevent slowing down.

[1301] Notification content: Sudden deceleration detected on X section of highway. Activate automatic brakes to prevent deceleration.

[1302] This will increase the safety and efficiency of self-driving vehicles.

[1303] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1304] Step 1:

[1305] The server collects historical driving data, weather information, traffic accident data, etc. from a database. The input is the database, and the output is raw data used to train the AI ​​model. Specifically, it uses SQL queries to retrieve the required data and saves it in CSV or JSON format.

[1306] Step 2:

[1307] The server trains an AI model based on the collected data. The input is the raw data collected in step 1, and the output is the trained AI model. Specifically, an anomaly detection model is built and trained using Python libraries such as Scikit-learn and TensorFlow.

[1308] Step 3:

[1309] The server receives sensor data and weather information sent from autonomous vehicles in real time. The input is real-time data, and the output is a dataset to be input into the AI ​​model. Specifically, the server receives real-time data via WebSocket or HTTP, preprocesses it, and formats it into a form that can be provided to the AI ​​model.

[1310] Step 4:

[1311] The server inputs the received real-time data into the AI ​​model for analysis. The input is a formatted data set, and the output is the analysis result (presence or absence of anomalies). Specifically, the predict method of the AI ​​model is called to detect abnormalities.

[1312] Step 5:

[1313] When the server detects an anomaly, it analyzes the details and generates a countermeasure. The input is the analysis result, and the output is a specific countermeasure and an anomaly notification message. Specifically, the generated anomaly notification and countermeasure are converted into JSON format and sent to the user's device.

[1314] Step 6:

[1315] The terminal displays the abnormality notification and countermeasures sent from the server visually and audibly. The input is the message from the server, and the output is the notification to the user. Specifically, the abnormality notification and countermeasures are displayed on the screen, and audio guidance is provided through the speaker.

[1316] Step 7:

[1317] The user checks the notified abnormal condition and takes appropriate action. The input is the notification from the terminal, and the output is the action taken and its results. Specifically, the user inputs feedback such as "Got it, apply brakes" into the terminal, which actually applies the brakes.

[1318] Step 8:

[1319] The device records the user's operations and the countermeasures taken, and sends the records to the server. The input is the user's feedback and operations, and the output is the recorded data. Specifically, the operations are saved in a log file and periodically uploaded to the server.

[1320] Step 9:

[1321] The server stores the recorded data in a database and displays it in a format that can be reviewed later. The input is the recorded data, and the output is the display items that can be reviewed by the user. Specifically, it inserts data into a database and visualizes the data in a web application or dashboard.

[1322] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1323] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[1324] System configuration

[1325] The system includes the following major components:

[1326] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[1327] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1328] 3. Server: Collects, processes and analyzes data.

[1329] 4. User terminal: An electronic device used by controllers and pilots to receive monitoring results and abnormality notifications.

[1330] 5. Emotion Engine: An engine that recognizes user emotions in real time and analyzes the data.

[1331] Program processing

[1332] server

[1333] 1. Data Collection:

[1334] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents, as well as user sentiment data.

[1335] 2. Training the AI ​​model:

[1336] The server trains an AI model based on the collected data. The AI ​​model learns patterns in the data to detect and predict anomalies. In particular, by including user emotional data, it becomes possible to provide emotion-based countermeasures for anomalies.

[1337] 3. Real-time data reception and analysis:

[1338] The server receives real-time operational data (flight information, aircraft status, weather information, etc.), provides this data to the AI ​​model, and analyzes the current operational status. The analysis results are used to monitor whether there are any operational anomalies.

[1339] 4. Detect anomalies and generate countermeasures:

[1340] The server uses an AI model to detect anomalies and identify their details (location, scope of impact, cause, etc.). It then generates countermeasures for the detected anomalies. Based on information from the emotion engine, it adjusts the priority of countermeasures according to the user's emotional state.

[1341] 5. Notification of abnormalities and response measures:

[1342] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[1343] Terminal

[1344] 1. Data display:

[1345] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[1346] 2. Warning Notice:

[1347] The terminal notifies the user of warnings and instructions from the server audibly and visually, records whether the user acknowledges the warning, and sends that information to the server.

[1348] 3. Receiving user input:

[1349] The terminal receives additional information and judgment results input by the user and transmits the data to the server in real time.

[1350] 4. Collecting Emotional Data:

[1351] The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[1352] 5. Data recording and archiving:

[1353] The device automatically records the user's operations and the countermeasures taken, and sends the recorded data to a server where it is stored in a database.

[1354] User

[1355] 1. Data review and situation assessment:

[1356] The user checks the information displayed on the device (such as abnormality notifications and countermeasures). Based on the displayed information, the user decides on the optimal countermeasure taking into account the actual operating conditions. The user can also refer to their own emotional data collected by the emotion engine.

[1357] 2. Enter and execute instructions:

[1358] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction. The details and results of the executed countermeasure are also input into the terminal, and the data is sent to the server.

[1359] 3. Warning confirmation and response:

[1360] The user checks the warnings and instructions sent from the device and takes the necessary action promptly. The device records the action and the situation, and sends that information to the server.

[1361] 4. Regular review and updates:

[1362] Users regularly check the monitoring data and warning information displayed on their devices, and take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1363] Specific examples

[1364] Server operation example

[1365] For example, when monitoring for abnormalities caused by bad weather, the server collects and learns from past bad weather data and analyzes it using an AI model. In addition, it also collects and learns data on the user's emotional state during past bad weather. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the results to the user's device. At this point, it also presents countermeasures prioritized according to the user's emotional state.

[1366] Example of terminal operation

[1367] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly, and the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[1368] Examples of user responses

[1369] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can refer to the analysis results of the emotion engine to select the most appropriate response. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, a user who receives an alert about a false intrusion on the runway can quickly implement a response and enter the results into the device. The emotion engine will evaluate the user's emotional state, and the results will also be used to improve the service.

[1370] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[1371] The processing flow will be explained below.

[1372] server

[1373] Step 1: Data collection

[1374] 1. The server collects past flight data, weather information, accident and incident history, and past emotion data from a database related to flight operations.

[1375] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model and emotion engine.

[1376] Step 2: Training the AI ​​model and emotion engine

[1377] 1. The server feeds the prepared dataset to the AI ​​model and emotion engine to train it for anomaly detection and emotion recognition.

[1378] 2. The server evaluates the trained AI model and emotion engine, and checks and adjusts its accuracy and processing power.

[1379] Step 3: Receive and analyze real-time data

[1380] 1. The server receives real-time operational data (flight information, aircraft status, weather information, etc.) and user emotion data.

[1381] 2. The server provides the received real-time data to the AI ​​model and emotion engine to analyze the current driving situation and the user's emotional state.

[1382] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation and monitors the user's emotional state.

[1383] Step 4: Detect anomalies and generate countermeasures

[1384] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[1385] 2. The server uses an emotion engine to evaluate the user's emotional state and reflects this in the anomaly detection results.

[1386] 3. The server generates countermeasures for the detected anomalies and adjusts the priority of the countermeasures according to the user's emotional state.

[1387] Step 5: Sending notification of abnormalities and countermeasures

[1388] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[1389] Terminal

[1390] Step 1: Data display

[1391] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[1392] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[1393] Step 2: Notification of warnings

[1394] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[1395] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[1396] Step 3: Receiving User Input

[1397] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[1398] Step 4: Collecting emotion data

[1399] 1. The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[1400] Step 5: Record and archive data

[1401] 1. The device automatically records user operations and the countermeasures taken.

[1402] 2. The terminal periodically sends the recorded data to the server and stores it in a database.

[1403] User

[1404] Step 1: Check the data and assess the situation

[1405] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[1406] 2. Based on the displayed information, the user decides on the best course of action to take, taking into account the actual driving situation and their own emotional state.

[1407] Step 2: Enter and execute instructions

[1408] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[1409] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[1410] Step 3: Check for warnings and take action

[1411] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[1412] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[1413] Step 4: Regularly check and update

[1414] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[1415] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1416] Specific examples

[1417] Server operation example

[1418] For example, when monitoring for abnormalities caused by bad weather, the server collects past bad weather data and analyzes it using an AI model. It also collects data on the user's emotional state during bad weather and analyzes it using an emotion engine. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of a specific flight and sends the results to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[1419] Example of terminal operation

[1420] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly. At this time, the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, along with the analyzed emotion data.

[1421] Examples of user responses

[1422] For example, a user who has received information about an approaching storm may decide to delay or cancel their flight, or to choose an alternative route. In this case, the user may refer to the analysis results of the emotion engine to select the optimal response. The selected response is then entered into the device, and the information is sent to the server and shared with relevant parties immediately. Furthermore, the user's emotional state is also recorded and analyzed, and this information is used to improve the service.

[1423] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[1424] Example 2

[1425] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1426] Conventional systems for detecting anomalies and providing countermeasures in aviation operations are prone to human error and place a heavy psychological burden on users. In particular, in emergency situations, the user's emotional state is not properly taken into account, which can result in the selection of the most appropriate countermeasure. This has led to a demand for improved safety and efficiency.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1428] In this invention, the server includes means for collecting past operation data and emotion data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures based on the detected anomalies and the user's emotional state, and means for transmitting the generated anomaly notification and countermeasures to the user terminal. This makes it possible to provide anomaly countermeasures that take the user's emotional state into consideration, thereby achieving improved safety and efficiency.

[1429] The "database" is an information collection point for storing and managing various data required by the system, such as past operation data and emotion data.

[1430] "Emotion data" is information that indicates the user's psychological state, and is data extracted from facial expressions, tone of voice, behavior, and the like.

[1431] An "AI model" is an algorithm that is trained based on collected data to detect anomalies and make predictions.

[1432] "Server" means a central control unit that collects, processes, and analyzes data and manages communications with user terminals.

[1433] A "user terminal" is an electronic device used by air traffic controllers and pilots to receive information and implement countermeasures.

[1434] "Anomaly detection" is the process of using AI models to analyze operational data and identify abnormal conditions or malfunctions.

[1435] "Countermeasures" refer to specific actions or instructions to be taken in response to detected abnormalities.

[1436] The "emotion engine" is software that analyzes the user's facial expressions and tone of voice, and collects and evaluates emotional data in real time.

[1437] An "abnormality notification" is a message or notification that notifies a user terminal of a detected abnormality and a countermeasure.

[1438] "Users" are human operators such as controllers and pilots who operate the system, review information, and implement countermeasures.

[1439] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[1440] The system includes the following major components:

[1441] 1. Database

[1442] It stores past operation data, weather information, accident and incident history, user emotion data, etc. To do this, it uses a commonly used relational database (e.g., MySQL).

[1443] 2. AI Model

[1444] This is an algorithm trained on database information to perform anomaly detection and prediction. It uses the Python TensorFlow library. In particular, by including user emotional data, it is possible to provide emotion-based countermeasures for anomalies.

[1445] 3. Server

[1446] It collects, processes, and analyzes data. It receives operational data (flight information, aircraft status, weather information, etc.) in real time and provides it to an AI model to analyze operational status. The server receives real-time data using WebSocket technology and sends the analysis results to the user's device.

[1447] 4. User Device

[1448] This is an electronic device used by air traffic controllers and pilots to receive monitoring results and abnormality notifications. The terminal displays information sent from the server in real time and issues audio and visual warnings. It also sends additional information and judgment results entered by the user to the server in real time, and uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data.

[1449] 5. Emotion Engine

[1450] This is an engine that recognizes user emotions in real time and analyzes the data. The emotion engine uses a camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. The collected data is sent to a server and used to generate countermeasures against abnormalities.

[1451] Specific examples

[1452] Server operation example

[1453] For example, when a server monitors for abnormalities caused by bad weather, it collects and learns from past bad weather data and analyzes it using an AI model. It also collects and learns data on the user's emotional state during past bad weather. If it analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the result to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[1454] Example of terminal operation

[1455] For example, if a device monitors a runway using a surveillance camera, it will display the video in real time and notify the user visually and audibly of any runway intrusion warnings from the server. At this time, the emotion engine analyzes the user's facial expressions and tone of voice, and when the user acknowledges the warning, that information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[1456] Examples of user responses

[1457] For example, if a user sees information about an approaching storm, they can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can select the optimal response, taking into account the analysis results of the emotion engine. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, if a user receives an alert about a false runway intrusion, they can quickly implement a response and enter the results into the device. The emotion engine will then evaluate the user's emotional state, and the results will also be used to improve the service.

[1458] Prompt Sentence Examples

[1459] "Please explain how the server collects historical severe weather data and operational data to train the AI ​​model. Also, please describe the process by which the weather information is analyzed in real time and anomaly response measures are generated based on the user's emotional state."

[1460] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1461] Program processing flow

[1462] server

[1463] Step 1:

[1464] Data collection

[1465] The server connects to the flight database to retrieve flight records, weather information, past accident and incident data, and user sentiment data. These data are retrieved using a MySQL database.

[1466] Input: Database connection information

[1467] Processing: Extract the required data from the database using SQL queries.

[1468] Output: A set of operational and emotional data

[1469] Step 2:

[1470] Training an AI model

[1471] The server uses the Python TensorFlow library to train the AI ​​model, which learns patterns to detect anomalies based on collected operational and emotional data.

[1472] Input: A set of operational data and emotion data

[1473] Processing: After data preprocessing (normalization, feature extraction, etc.), training is performed using an AI model.

[1474] Output: A trained AI model

[1475] Step 3:

[1476] Receiving and analyzing real-time data

[1477] The server periodically receives real-time data (flight information, aircraft status, weather data, etc.) sent from the aircraft using WebSocket technology, and provides the received data to an AI model to analyze the current operating status.

[1478] Input: Real-time data

[1479] Processing: The received data is fed into an AI model for analysis to determine whether there are any anomalies.

[1480] Output: Current operating status and abnormality detection results

[1481] Step 4:

[1482] Detecting anomalies and generating countermeasures

[1483] The server uses an AI model to detect anomalies. If an anomaly is detected, it identifies the details (location of occurrence, scope of impact, cause, etc.), adjusts the priority of countermeasures based on data from the emotion engine, and then generates countermeasures.

[1484] Input: Anomaly detection results and emotion data

[1485] Action: Generate and prioritize countermeasures based on detected anomalies.

[1486] Output: Abnormal notification and countermeasures

[1487] Step 5:

[1488] Sending abnormality notifications and countermeasures

[1489] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[1490] Input: Abnormal notification and countermeasures

[1491] Processing: Convert the notification data into an appropriate format and send it to the user terminal.

[1492] Output: Abnormal notification to user terminal

[1493] Terminal

[1494] Step 1:

[1495] Data Display

[1496] The terminal displays the monitoring results and abnormality notifications sent from the server in real time, using an interface that uses a GUI library.

[1497] Input: Abnormal notification and countermeasures

[1498] Processing: Updates various display elements of the GUI.

[1499] Output: Displayed abnormality notification and countermeasures

[1500] Step 2:

[1501] Warning Notification

[1502] The terminal will notify the user of warnings and instructions from the server audibly and visually, including playing audio files and visual alerts.

[1503] Input: Error notification and warning information

[1504] Action: Plays an audio and displays a visual alert.

[1505] Output: Audio and visual warnings

[1506] Step 3:

[1507] Receiving User Input

[1508] The terminal receives input from the user and transmits the data to the server in real time, either via a touchscreen or keyboard input.

[1509] Input: User-entered data

[1510] Processing: Sends the received user input to the server.

[1511] Output: Send input data to the server

[1512] Step 4:

[1513] Collecting Emotional Data

[1514] The device uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data, using a camera and microphone.

[1515] Input: User's facial expression data and voice data

[1516] Processing: Emotion data is generated using facial expression analysis algorithms and voice analysis algorithms.

[1517] Output: Emotion data

[1518] Step 5:

[1519] Data recording and archiving

[1520] The device automatically records the user's actions and the countermeasures taken, and transmits the data to the server using local storage and network communication.

[1521] Input: User operation log and execution details

[1522] Processing: The log data is stored in a format and sent to the server.

[1523] Output: Send log data to the server

[1524] User

[1525] Step 1:

[1526] Data confirmation and situation assessment

[1527] The user checks the information displayed on the device, determines the best course of action based on the operational status, and takes appropriate action based on the displayed information.

[1528] Input: Abnormality notification and countermeasures displayed on the terminal

[1529] Processing: Analyze the displayed information and make decisions based on the situation.

[1530] Output: Decision results and action selection

[1531] Step 2:

[1532] Entering and executing instructions

[1533] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction.

[1534] Input: Workaround

[1535] Processing: Executes the input instructions and inputs the results to the terminal.

[1536] Output: Input data and execution results to the terminal

[1537] Step 3:

[1538] Checking and responding to warnings

[1539] The user checks the warnings and instructions from the terminal and takes appropriate action promptly.

[1540] Input: Warning notices and instructions

[1541] Action: Assess the situation and take the most appropriate action.

[1542] Output: Countermeasure execution results

[1543] Step 4:

[1544] Regular review and updates

[1545] Users regularly check the monitoring data and warning information displayed on their devices and take preventive measures or make advance preparations as necessary.

[1546] Input: Monitoring data and warning information

[1547] Processing: Regularly review the data and take any necessary action.

[1548] Output: Preventive measures taken and preparatory results

[1549] (Application example 2)

[1550] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1551] Conventional autonomous vehicle traffic management systems can detect anomalies and generate countermeasures in real time, but they cannot consider the emotional state of passengers and drivers. As a result, psychological burden increases and appropriate responses are delayed. Therefore, in order to improve safety and efficiency during operation, it is necessary to analyze the user's emotional state in real time and provide appropriate countermeasures.

[1552] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past operation data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures for the detected anomaly, means for transmitting the generated anomaly notification and countermeasures to the user terminal, means for analyzing the user's emotional state using an emotion engine, and means for adjusting the priority of the countermeasures based on the analyzed emotional state. This makes it possible to provide anomaly countermeasures according to the user's emotional state.

[1553] The "database" is an information management system for storing past operational data, weather information, accident and incident history, and user emotional data.

[1554] "Operation data" refers to information about the operating status of an autonomous vehicle, including, for example, flight records, weather information, and vehicle sensor data.

[1555] An "AI model" is an algorithm that is trained on information from a database to perform anomaly detection and prediction.

[1556] "Anomaly detection" is the process of using AI models to identify unusual behaviors or situations in operational data.

[1557] "Countermeasures" are specific measures or action plans for the detected abnormality, and are intended to ensure safety and improve operational efficiency.

[1558] A "user terminal" is a device used by controllers and drivers, and is an electronic device for receiving monitoring results and abnormality notifications.

[1559] The "emotion engine" is a system that analyzes the user's emotional state in real time and reflects that data in operational management.

[1560] "Priority" is an index that indicates the importance and urgency of implementing an abnormality response measure, and is adjusted based on the emotional state of the user.

[1561] An "abnormality notification" is a message for transmitting information about a detected abnormality to a user terminal.

[1562] "Analysis" is the process of identifying and understanding situations and patterns based on collected data.

[1563] The present invention combines an emotion engine with a safe driving management system for autonomous vehicles, making it possible to provide countermeasures against abnormalities that take into account the emotional state of the user. A specific embodiment of the system for implementing the present invention will be described below.

[1564] System configuration

[1565] The system includes the following major components:

[1566] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[1567] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1568] 3. Server: Collects, processes and analyzes data.

[1569] 4. User terminal: An electronic device used by the driver or passenger to receive monitoring results and abnormality notifications.

[1570] 5. Emotion engine: Analyzes user emotions in real time and reflects that data in operation management.

[1571] Program processing

[1572] server

[1573] The server collects historical driving data (e.g., speed, location, and surrounding obstacle information) from the driving database. This data includes vehicle sensor data, real-time weather data, traffic condition data, and user emotion data. Based on the collected data, the server trains an AI model to detect and predict anomalies. Furthermore, the server receives real-time driving data, detects anomalies using the AI ​​model, and generates countermeasures. The server adjusts the priority of countermeasures based on the emotion data analyzed by the emotion engine.

[1574] User terminal

[1575] The user device receives the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server and notifies the user visually and audibly. The device also uses a camera and smart glasses to analyze the user's facial expressions and tone of voice to collect emotional data. This allows the user's emotional state to be sent to the server in real time and used to adjust the priority of countermeasures to anomalies.

[1576] User

[1577] The user checks the information displayed on the device (such as abnormality notifications and countermeasures) and selects and executes the most appropriate countermeasure for the situation. In doing so, the user also takes into account their own emotional data analyzed by the emotion engine. For example, after checking information about an approaching storm, the user can decide whether to delay or cancel a flight or select an alternative route. The countermeasures taken are entered into the device and recorded and analyzed on the server.

[1578] Specific examples

[1579] For example, when an autonomous vehicle detects abnormal data during bad weather, the system predicts the abnormality based on past weather data and the user's emotional state at the time. If passenger anxiety increases due to the emotion engine, the system can quickly provide high-priority countermeasures such as delays or route changes. After the user confirms and acts on the warning, the information is sent to a server and stored in a database.

[1580] Example prompts to input to the generative AI model

[1581] Get up-to-date traffic data, calculate safe and efficient routes based on weather data and vehicle sensor data, and take into account the emotional state (e.g., stress, relief) of the passengers and driver in the car.

[1582] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1583] Step 1:

[1584] The server collects historical driving data from a database. This data includes vehicle sensor data (speed, location, obstacle information, etc.), weather data, traffic condition data, and user emotion data. This creates a dataset that provides a comprehensive understanding of driving conditions. The collected data is used to train AI models for anomaly detection and prediction.

[1585] Input: Historical driving data, sensor data, weather data, traffic data, user emotion data

[1586] Output: Collected dataset

[1587] Step 2:

[1588] The server trains the AI ​​model based on the collected data. This training process learns patterns in the data and builds algorithms capable of detecting and predicting anomalies. In particular, by including user emotional data, it is possible to generate countermeasures to anomalies that take into account the user's emotional state.

[1589] Input: Collected dataset

[1590] Output: A trained AI model

[1591] Step 3:

[1592] The server receives real-time operational data, including the vehicle's current sensor data, real-time weather information, and traffic data, which is then fed into an AI model to analyze the current operational situation.

[1593] Input: Real-time operation data, sensor data, weather information, traffic condition data

[1594] Output: Parsed operational status data

[1595] Step 4:

[1596] The server uses an AI model to detect anomalies. It then identifies the details of the detected anomaly (location, scope of impact, cause, etc.) and generates countermeasures. It then adjusts the priority of countermeasures based on the user's emotional data collected by the emotion engine.

[1597] Input: Analyzed driving status data, real-time emotion data

[1598] Output: Coordinated response measures, anomaly notification

[1599] Step 5:

[1600] The server sends the generated abnormality notification and countermeasures to the user terminal. The user terminal receives this and notifies the user visually and audibly. The user checks the information displayed on the terminal and selects and executes the appropriate countermeasures.

[1601] Input: Coordinated action, anomaly notification

[1602] Output: User's action selection information

[1603] Step 6:

[1604] The user device uses a camera and smart glasses to analyze the user's facial expressions and tone of voice, collecting emotional data in real time. This emotional data is sent to a server and used to prioritize countermeasures against abnormalities.

[1605] Input: Real-time video and audio data of the user

[1606] Output: Real-time emotion data

[1607] Step 7:

[1608] The user enters the details and results of the selected action into the device and sends the data to the server, which stores this information in a database and uses it for future analysis and service improvement.

[1609] Input: User-entered details of countermeasures, execution results

[1610] Output: Details of saved countermeasures, execution result data

[1611] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1613] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1614] [Fourth embodiment]

[1615] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1616] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1617] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1618] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1619] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1620] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1621] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1622] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1623] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1624] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1625] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1626] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1627] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1628] This invention relates to an air navigation support system, which utilizes AI to support monitoring, guidance, and decision-making in air operations, with the aim of reducing human error and improving the safety and efficiency of air operations.

[1629] System configuration

[1630] The system includes the following major components:

[1631] 1. Database: Stores past operation data, weather information, accident and incident history, etc.

[1632] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1633] 3. Server: Collects, processes and analyzes data.

[1634] 4. User terminal: Electronic equipment used by controllers and pilots to receive monitoring results and abnormality notifications.

[1635] Program processing

[1636] server

[1637] 1. Data Collection:

[1638] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents.

[1639] 2. Training the AI ​​model:

[1640] The server trains an AI model based on the collected data, which learns patterns in the data to detect and predict anomalies.

[1641] 3. Real-time data reception and analysis:

[1642] The server receives real-time operational data, such as current weather information, flight status, and surveillance camera footage, and provides this data to the AI ​​model, which monitors and analyzes it for any abnormalities.

[1643] 4. Anomaly detection and notification:

[1644] When an anomaly is detected by the AI ​​model, the server analyzes the details and generates a countermeasure, then sends the generated anomaly notification and countermeasure to the user's device.

[1645] Terminal

[1646] 1. Data display:

[1647] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[1648] 2. Receiving user input:

[1649] The terminal receives additional information and judgment results input by the user and transmits them to the server, thereby updating the information in real time.

[1650] 3. Warning Notice:

[1651] The terminal notifies the user of warnings and instructions from the server by voice or visually, and records whether the user acknowledges the warning or not, and sends the record to the server.

[1652] 4. Data recording and archiving:

[1653] The device automatically records the user's actions and the countermeasures taken, sends the data to a server, stores it in a database, and displays it in a format that can be reviewed later.

[1654] User

[1655] 1. Data verification and judgment:

[1656] The user checks the information displayed on the device (for example, an abnormality notification or recommended countermeasures). Based on the displayed information, the user decides on the most appropriate countermeasure in light of the actual situation.

[1657] 2. Enter and execute instructions:

[1658] The user inputs the response they have decided on into the terminal and executes the instructions. By inputting additional information and comments and sending them to the server, more precise responses can be made.

[1659] 3. Warning confirmation and response:

[1660] The user checks the warnings and instructions (for example, an anomaly detection notification) from the terminal and takes the necessary action. The results of the action and the details of the action are entered into the terminal and sent to the server.

[1661] 4. Regular checks:

[1662] Users regularly check the monitoring data and warning information displayed on their devices. Even if no abnormalities are detected, they can take appropriate precautions and check for countermeasures according to the situation.

[1663] Specific examples

[1664] Server operation example

[1665] For example, when monitoring for bad weather, the server collects data on bad weather over the past 10 years and uses an AI model to learn what weather conditions will affect flights. When the server analyzes weather information in real time and detects an approaching storm, it determines whether a particular flight is safe and sends the result to the user's device.

[1666] Example of terminal operation

[1667] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the images transmitted in real time and notifies the user visually and audibly of runway intrusion warnings from the server. After the user performs a confirmation action, the information is sent to the server.

[1668] Examples of user responses

[1669] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight or choose an alternative route, and input their response into their device. The information is then sent to the server and shared immediately with relevant parties. Similarly, a user who receives a false runway intrusion alert can quickly implement a response, input the results into their device, and send them to the server.

[1670] As described above, the system of the present invention utilizes AI to support rapid and appropriate responses while minimizing human error in order to improve the safety and efficiency of operations.

[1671] The processing flow will be explained below.

[1672] server

[1673] Step 1: Data collection

[1674] 1. The server collects historical flight data from a database of flight operations.

[1675] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model.

[1676] Step 2: Training the AI ​​model

[1677] 1. The server feeds the prepared dataset to the AI ​​model and trains the model for anomaly detection and prediction.

[1678] 2. The server evaluates the trained AI model and checks and adjusts its accuracy and processing power.

[1679] Step 3: Receive and analyze real-time data

[1680] 1. The server receives operational data (flight information, aircraft status, weather information, etc.) in real time.

[1681] 2. The server provides the received real-time data to the AI ​​model and analyzes the current operating conditions.

[1682] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation.

[1683] Step 4: Detect anomalies and generate countermeasures

[1684] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[1685] 2. The server generates countermeasures for the detected anomalies.

[1686] Step 5: Sending notification and countermeasures

[1687] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[1688] Terminal

[1689] Step 1: Data display

[1690] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[1691] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[1692] Step 2: Notification of warnings

[1693] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[1694] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[1695] Step 3: Receiving input from the user

[1696] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[1697] Step 4: Record and archive data

[1698] 1. The device automatically records user operations and the countermeasures taken.

[1699] 2. The device sends the recorded data to the server and stores it in a database.

[1700] User

[1701] Step 1: Check the data and assess the situation

[1702] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[1703] 2. Based on the displayed information, the user decides on the best course of action to take in light of the actual operating conditions.

[1704] Step 2: Enter and execute instructions

[1705] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[1706] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[1707] Step 3: Check for warnings and take action

[1708] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[1709] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[1710] Step 4: Regularly check and update

[1711] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[1712] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1713] Example 1

[1714] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1715] To improve the safety and efficiency of air travel, there is a need for real-time anomaly detection and rapid response while minimizing human error. However, conventional systems lack the accuracy to process large amounts of operational data and detect anomalies, making it difficult to respond in real time. This can lead to reduced safety and operational delays, so a new system is needed to solve these issues.

[1716] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1717] In this invention, the server includes means for collecting past flight data from a database, means for training a machine learning model for anomaly detection based on the collected data, means for receiving flight-related data in real time and detecting anomalies using the machine learning model, means for generating countermeasures for the detected anomalies, and means for transmitting the generated anomaly notification and countermeasures to the operation terminal, thereby enabling real-time anomaly detection and rapid response, thereby improving the safety and efficiency of airline operations.

[1718] A "database" is a system designed for the purpose of organizing and storing information, and centrally manages past operational data, weather information, accident and incident history, and more.

[1719] A "machine learning model" is an algorithm that learns patterns based on large amounts of data and makes predictions and detects anomalies in new data.

[1720] "Flight-related data" refers to all information related to aircraft operations, including real-time flight status, weather information, and surveillance camera footage.

[1721] "Anomaly detection" is the process of identifying abnormal situations that deviate from normal operating conditions, allowing for prompt countermeasures to be taken.

[1722] "Countermeasures" refer to actions or measures taken in response to detected abnormalities, with the aim of solving problems or avoiding risks.

[1723] An "operation terminal" is an electronic device used by air traffic controllers and pilots, which notifies users of abnormalities, displays countermeasures, and receives user input.

[1724] A "communication device" is hardware and software for transmitting and receiving data, and is used to exchange real-time data between a server and a terminal.

[1725] "Data Processing Device" means a collection of hardware and software used to process and analyze collected data for anomaly detection and countermeasure generation.

[1726] This invention is a system for improving the safety and efficiency of airline operations, and aims to reduce human error by using machine learning. This system is built through collaboration between servers, terminals, and users, and enables real-time anomaly detection and response.

[1727] server

[1728] The server collects and processes large amounts of operational data, trains machine learning models, and receives and analyzes real-time data. Specifically, it uses the following hardware and software:

[1729] Hardware: High-performance server

[1730] Software: MySQL (database management), TensorFlow (building and training machine learning models), Kafka (receiving and processing real-time data)

[1731] The server collects past operation data from a MySQL database and uses the collected data to train a machine learning model using TensorFlow. The operation data received in real time is processed using Kafka to detect anomalies. If an anomaly is detected, the details are analyzed and a countermeasure is generated. This is all done automatically, and the generated anomaly notification and countermeasure are sent to the terminal.

[1732] For example, to train a forecasting model for severe weather, the server uses TensorFlow to learn how weather conditions affect flights based on 10 years of weather data. If a storm is approaching, the server immediately determines whether a particular flight is safe and notifies the device of the result.

[1733] Terminal

[1734] The terminal visually displays the data sent from the server and supports the user. The following hardware and software are used:

[1735] Hardware: High-resolution display, speakers

[1736] Software: HTML / CSS (dashboard display), JavaScript (asynchronous communication)

[1737] The terminal immediately displays the abnormality notification and countermeasures sent from the server and gives a voice notification. Information entered by the user and the judgment results are sent to the server in real time using JavaScript. The terminal also records operation logs, sends them to the server and stores them in a database.

[1738] For example, in the case of runway monitoring using a surveillance camera, the terminal displays the surveillance video transmitted in real time and notifies the user of runway intrusion warnings visually and audibly. After the user performs a confirmation action, the information is sent to the server.

[1739] User

[1740] The user can make a decision based on the information displayed on the device and take the necessary action. Specific examples of operations are as follows:

[1741] The user checks the abnormality notification displayed on the terminal and decides on the appropriate countermeasure depending on the situation. The user inputs the countermeasure into the terminal and executes it. If necessary, the user can also input additional information or comments and send them to the server in real time.

[1742] For example, if a user sees information about an approaching storm, they can decide whether to delay, cancel, or reroute their flight and input their response plan into their device. The information is then sent to the server and shared with relevant parties immediately. If a false runway intrusion alarm is received, the user can quickly take action and input the results into their device and send them to the server.

[1743] Prompt Sentence Examples

[1744] "Learn how storms affect your flight using weather data from the past decade."

[1745] "Please create a system that provides visual and audio warnings for runway intrusions."

[1746] "Explain how flights will be delayed, canceled, or rerouted based on approaching storms."

[1747] This system significantly improves the safety and efficiency of air travel by linking servers, terminals, and users to detect and respond to abnormalities in real time.

[1748] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1749] Step 1:

[1750] The server collects historical flight data from the database. Specifically, it connects to the MySQL database and executes the SQL query "SELECT FROM flight_data WHERE date BETWEEN '2010-01-01' AND '2020-12-31';". The input is the SQL query, and the output is data on past flight records, weather information, and accidents and incidents. This output data is used as training data to build an anomaly detection model.

[1751] Step 2:

[1752] The server trains a machine learning model based on the collected data. Data preprocessing involves filling in missing values ​​and normalizing the data. Specifically, the model is trained using TensorFlow using "model.fit(training_data, labels, epochs=50)". The input is the preprocessed data, and the output is a trained anomaly detection model.

[1753] Step 3:

[1754] The server receives flight-related data in real time. Specifically, it uses Kafka to receive data via "consumer.poll(100)" and processes it as real-time data. The input is real-time flight status, weather information, and surveillance camera footage, and the output is packets containing this data.

[1755] Step 4:

[1756] The server inputs real-time data into a machine learning model for analysis. Specifically, it inputs the received data into "model.predict(new_flight_data)" to detect anomalies. The input is real-time data, and the output is the prediction result (presence or absence of anomalies).

[1757] Step 5:

[1758] When an anomaly is detected, the server analyzes the details and generates countermeasures. Specifically, it runs "analyze_anomaly(predictions)" to analyze the details of the anomaly and generate countermeasures. The input is the prediction result, and the output is an anomaly notification including countermeasures.

[1759] Step 6:

[1760] The server sends the generated abnormality notification and countermeasures to the operation terminal. Specifically, it uses Kafka to send notifications as follows: "producer.send('alerts', alert_message)". The input is the abnormality notification and countermeasures, and the output is a notification message to the operation terminal.

[1761] Step 7:

[1762] The terminal displays the data sent from the server on the screen. Specifically, a warning message is displayed on a dashboard created with HTML / CSS using "document.getElementById('alert').innerHTML = alert_message;". The input is the notification message from the server, and the output is the abnormality notification and countermeasures displayed on the terminal screen.

[1763] Step 8:

[1764] The device receives additional information and decision results entered by the user and sends them to the server via asynchronous communication. Specifically, it uses the JavaScript "fetch" API, using "fetch(' / submit-decisions', { method: 'POST', body: JSON.stringify(user_decision)})". The input is the user's decision result, and the output is the data to be sent to the server.

[1765] Step 9:

[1766] The terminal notifies the user of warnings and instructions from the server visually and audibly. Specifically, it uses the browser's notification API to display a warning with "new Notification('Warning', { body: 'An abnormality has been detected on the runway.'})". The input is the warning message from the server, and the output is the notification to the user.

[1767] Step 10:

[1768] The user makes a decision based on the information displayed on the terminal and takes the necessary action. Specifically, the user checks the warning message and inputs the action, such as flight delay, cancellation, or rerouting, into the terminal. The input is the information displayed on the terminal, and the output is the action taken and its results.

[1769] Step 11:

[1770] User operations and the details of the countermeasures taken are recorded on the terminal, and the data is sent to the server and stored in a database. Specifically, the operation log is collected in JSON format and sent to the server using "fetch(' / log-actions', { method: 'POST', body: JSON.stringify(action_log)})". The input is the user operation log, and the output is the recorded data stored in the database.

[1771] (Application example 1)

[1772] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1773] Conventional autonomous vehicle systems are inadequate in their real-time response to changes in traffic and weather conditions, making it difficult to detect abnormalities and generate appropriate countermeasures. This raises concerns about traffic accidents and reduced safety. Furthermore, they lack the ability to quickly incorporate user feedback and improve system accuracy. There is a need for a system that can solve these issues and improve the safety and efficiency of autonomous vehicles.

[1774] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1775] In this invention, the server includes: means for collecting past operation data from a database; means for training an AI model for anomaly detection based on the collected data; means for receiving operation data in real time and detecting anomalies using the AI ​​model; means for generating countermeasures for the detected anomalies; means for transmitting the generated anomaly notification and countermeasures to a user terminal; means for collecting real-time sensor data and weather information from the autonomous vehicle; means for analyzing the collected data using the AI ​​model to detect abnormal conditions; means for displaying the anomaly notification and countermeasures visually and audibly on the user terminal; means for receiving user input and providing real-time feedback to the system; means for recording user operations and anomaly response details and saving the records in a database; and means for displaying the saved record data in a format that can be reviewed later. This makes it possible to monitor changes in traffic and weather conditions in real time while the autonomous vehicle is operating, quickly detect abnormal conditions, and automatically generate appropriate countermeasures, thereby improving safety and efficiency.

[1776] A "database" is a storage device that accumulates and stores past operation data, weather information, traffic accident data, and the like.

[1777] "Operation data" refers to all information that indicates the vehicle's operating status, such as various sensor data, camera footage, location information, and speed information from an autonomous vehicle.

[1778] An "AI model" is an algorithm trained to detect and predict anomalies based on collected data.

[1779] "Sensor data" refers to real-time operational information acquired by various sensors (e.g., cameras, lidar, radar) installed in autonomous vehicles.

[1780] "Weather information" refers to information about weather conditions during operation, such as wind speed, rainfall, and temperature.

[1781] An "abnormal state" is an unusual situation detected during operation (e.g., sudden deceleration, lane departure, approaching obstacle, etc.).

[1782] "Countermeasures" are specific actions or measures that should be taken in response to a detected abnormal condition.

[1783] A "user terminal" is a device (for example, an in-car display or a smartphone) that displays information sent from a server and receives user input.

[1784] "Feedback" is real-time input, such as additional information or decisions provided by the user, that is used to improve the system.

[1785] "Voice notification" is a means of conveying warnings and instructions to the user by voice.

[1786] A "visual display" is a means of displaying information using text and graphics on a user terminal display.

[1787] This invention is a system that improves the operational safety and efficiency of autonomous vehicles. It uses AI to monitor changes in traffic and weather conditions in real time, quickly detects abnormal conditions, and automatically generates appropriate countermeasures.

[1788] System Configuration

[1789] The system includes the following major components:

[1790] server

[1791] The server has the following roles:

[1792] 1. Database (accumulation of past operation data):

[1793] The database will store past driving data, weather information, traffic accident data, etc. This data is collected from autonomous vehicles in operation.

[1794] 2. Training the AI ​​model:

[1795] The server uses the collected data to train an AI model, which learns patterns of abnormal conditions and enables real-time anomaly detection.

[1796] 3. Real-time data reception and analysis:

[1797] The server receives sensor data and weather information sent from the autonomous vehicle in real time and analyzes it using an AI model.

[1798] 4. Detect anomalies and generate countermeasures:

[1799] If an anomaly is detected, the details are analyzed and countermeasures are generated, providing specific measures to keep the autonomous vehicle operating safely.

[1800] 5. Sending abnormality notification and countermeasures to user devices:

[1801] A notification of the detected abnormality and the generated countermeasure are sent to a user terminal inside the vehicle.

[1802] User devices (displays and smartphones inside autonomous vehicles)

[1803] The user terminal provides the following functions:

[1804] 1. Data display:

[1805] Displays abnormal notifications and countermeasures sent from the server visually and audibly.

[1806] 2. Receiving user input:

[1807] Receive feedback and additional information from the user and send that information to the server.

[1808] 3. Warning notice:

[1809] If an abnormality is detected, the user will be notified with a visual and audio warning.

[1810] 4. Data recording and archiving:

[1811] The user's operations and responses are recorded and the data is saved in a database.

[1812] User

[1813] The user roles are as follows:

[1814] 1. Check the data and take action:

[1815] Check for abnormality notifications and recommended actions, and take action as necessary.

[1816] 2. Providing Feedback:

[1817] Provide real-time feedback to the system to improve data accuracy.

[1818] Specific examples

[1819] For example, if an autonomous vehicle detects sudden deceleration while traveling on a highway, the system operates as follows:

[1820] 1. The server analyzes sudden deceleration data on the highway using an AI model and detects it as an abnormal condition.

[1821] 2. The server generates a countermeasure for the detected abnormal condition, such as "activate the automatic brakes to prevent deceleration."

[1822] 3. The generated abnormality notification and countermeasures are sent to the user terminal.

[1823] 4. The user device visually displays the abnormal condition and countermeasures on the in-car display and also notifies the driver audibly.

[1824] 5. The user acknowledges the notification and provides appropriate feedback to the system (e.g., "Got it, brake on").

[1825] 6. User actions are recorded and saved in a format that can be reviewed at a later date.

[1826] Example prompt sentence:

[1827] For example, if an AI model detects a sudden deceleration on a highway and generates a countermeasure, it might generate a prompt like this:

[1828] Abnormal condition: sudden deceleration

[1829] Location information: Expressway X section

[1830] Recommended action: Activate the automatic brakes to prevent slowing down.

[1831] Notification content: Sudden deceleration detected on X section of highway. Activate automatic brakes to prevent deceleration.

[1832] This will increase the safety and efficiency of self-driving vehicles.

[1833] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1834] Step 1:

[1835] The server collects historical driving data, weather information, traffic accident data, etc. from a database. The input is the database, and the output is raw data used to train the AI ​​model. Specifically, it uses SQL queries to retrieve the required data and saves it in CSV or JSON format.

[1836] Step 2:

[1837] The server trains an AI model based on the collected data. The input is the raw data collected in step 1, and the output is the trained AI model. Specifically, an anomaly detection model is built and trained using Python libraries such as Scikit-learn and TensorFlow.

[1838] Step 3:

[1839] The server receives sensor data and weather information sent from autonomous vehicles in real time. The input is real-time data, and the output is a dataset to be input into the AI ​​model. Specifically, the server receives real-time data via WebSocket or HTTP, preprocesses it, and formats it into a form that can be provided to the AI ​​model.

[1840] Step 4:

[1841] The server inputs the received real-time data into the AI ​​model for analysis. The input is a formatted data set, and the output is the analysis result (presence or absence of anomalies). Specifically, the predict method of the AI ​​model is called to detect abnormalities.

[1842] Step 5:

[1843] When the server detects an anomaly, it analyzes the details and generates a countermeasure. The input is the analysis result, and the output is a specific countermeasure and an anomaly notification message. Specifically, the generated anomaly notification and countermeasure are converted into JSON format and sent to the user's device.

[1844] Step 6:

[1845] The terminal displays the abnormality notification and countermeasures sent from the server visually and audibly. The input is the message from the server, and the output is the notification to the user. Specifically, the abnormality notification and countermeasures are displayed on the screen, and audio guidance is provided through the speaker.

[1846] Step 7:

[1847] The user checks the notified abnormal condition and takes appropriate action. The input is the notification from the terminal, and the output is the action taken and its results. Specifically, the user inputs feedback such as "Got it, apply brakes" into the terminal, which actually applies the brakes.

[1848] Step 8:

[1849] The device records the user's operations and the countermeasures taken, and sends the records to the server. The input is the user's feedback and operations, and the output is the recorded data. Specifically, the operations are saved in a log file and periodically uploaded to the server.

[1850] Step 9:

[1851] The server stores the recorded data in a database and displays it in a format that can be reviewed later. The input is the recorded data, and the output is the display items that can be reviewed by the user. Specifically, it inserts data into a database and visualizes the data in a web application or dashboard.

[1852] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1853] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[1854] System configuration

[1855] The system includes the following major components:

[1856] 1. Database: Stores past operation data, weather information, accident and incident history, user emotion data, etc.

[1857] 2. AI model: An algorithm trained on information from a database to perform anomaly detection and prediction.

[1858] 3. Server: Collects, processes and analyzes data.

[1859] 4. User terminal: An electronic device used by controllers and pilots to receive monitoring results and abnormality notifications.

[1860] 5. Emotion Engine: An engine that recognizes user emotions in real time and analyzes the data.

[1861] Program processing

[1862] server

[1863] 1. Data Collection:

[1864] The server collects historical flight data from a flight database, including flight records, weather information, and data on past accidents and incidents, as well as user sentiment data.

[1865] 2. Training the AI ​​model:

[1866] The server trains an AI model based on the collected data. The AI ​​model learns patterns in the data to detect and predict anomalies. In particular, by including user emotional data, it becomes possible to provide emotion-based countermeasures for anomalies.

[1867] 3. Real-time data reception and analysis:

[1868] The server receives real-time operational data (flight information, aircraft status, weather information, etc.), provides this data to the AI ​​model, and analyzes the current operational status. The analysis results are used to monitor whether there are any operational anomalies.

[1869] 4. Detect anomalies and generate countermeasures:

[1870] The server uses an AI model to detect anomalies and identify their details (location, scope of impact, cause, etc.). It then generates countermeasures for the detected anomalies. Based on information from the emotion engine, it adjusts the priority of countermeasures according to the user's emotional state.

[1871] 5. Notification of abnormalities and response measures:

[1872] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[1873] Terminal

[1874] 1. Data display:

[1875] The terminal displays the monitoring results (notification of abnormalities, countermeasures, etc.) sent from the server on its screen, presenting the information in a visually easy-to-understand format.

[1876] 2. Warning Notice:

[1877] The terminal notifies the user of warnings and instructions from the server audibly and visually, records whether the user acknowledges the warning, and sends that information to the server.

[1878] 3. Receiving user input:

[1879] The terminal receives additional information and judgment results input by the user and transmits the data to the server in real time.

[1880] 4. Collecting Emotional Data:

[1881] The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[1882] 5. Data recording and archiving:

[1883] The device automatically records the user's operations and the countermeasures taken, and sends the recorded data to a server where it is stored in a database.

[1884] User

[1885] 1. Data review and situation assessment:

[1886] The user checks the information displayed on the device (such as abnormality notifications and countermeasures). Based on the displayed information, the user decides on the optimal countermeasure taking into account the actual operating conditions. The user can also refer to their own emotional data collected by the emotion engine.

[1887] 2. Enter and execute instructions:

[1888] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction. The details and results of the executed countermeasure are also input into the terminal, and the data is sent to the server.

[1889] 3. Warning confirmation and response:

[1890] The user checks the warnings and instructions sent from the device and takes the necessary action promptly. The device records the action and the situation, and sends that information to the server.

[1891] 4. Regular review and updates:

[1892] Users regularly check the monitoring data and warning information displayed on their devices, and take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1893] Specific examples

[1894] Server operation example

[1895] For example, when monitoring for abnormalities caused by bad weather, the server collects and learns from past bad weather data and analyzes it using an AI model. In addition, it also collects and learns data on the user's emotional state during past bad weather. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the results to the user's device. At this point, it also presents countermeasures prioritized according to the user's emotional state.

[1896] Example of terminal operation

[1897] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly, and the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[1898] Examples of user responses

[1899] For example, a user who has received information about an approaching storm can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can refer to the analysis results of the emotion engine to select the most appropriate response. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, a user who receives an alert about a false intrusion on the runway can quickly implement a response and enter the results into the device. The emotion engine will evaluate the user's emotional state, and the results will also be used to improve the service.

[1900] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[1901] The processing flow will be explained below.

[1902] server

[1903] Step 1: Data collection

[1904] 1. The server collects past flight data, weather information, accident and incident history, and past emotion data from a database related to flight operations.

[1905] 2. The server organizes and classifies the collected data and prepares it as a training dataset for the AI ​​model and emotion engine.

[1906] Step 2: Training the AI ​​model and emotion engine

[1907] 1. The server feeds the prepared dataset to the AI ​​model and emotion engine to train it for anomaly detection and emotion recognition.

[1908] 2. The server evaluates the trained AI model and emotion engine, and checks and adjusts its accuracy and processing power.

[1909] Step 3: Receive and analyze real-time data

[1910] 1. The server receives real-time operational data (flight information, aircraft status, weather information, etc.) and user emotion data.

[1911] 2. The server provides the received real-time data to the AI ​​model and emotion engine to analyze the current driving situation and the user's emotional state.

[1912] 3. Based on the analysis results, the server monitors whether there are any abnormalities in operation and monitors the user's emotional state.

[1913] Step 4: Detect anomalies and generate countermeasures

[1914] 1. The server uses an AI model to detect anomalies and identify their details (location of occurrence, scope of impact, cause, etc.).

[1915] 2. The server uses an emotion engine to evaluate the user's emotional state and reflects this in the anomaly detection results.

[1916] 3. The server generates countermeasures for the detected anomalies and adjusts the priority of the countermeasures according to the user's emotional state.

[1917] Step 5: Sending notification of abnormalities and countermeasures

[1918] 1. The server sends the generated abnormality notification and countermeasures to the user terminal.

[1919] Terminal

[1920] Step 1: Data display

[1921] 1. The terminal displays the monitoring results (anomaly notifications, countermeasures, etc.) sent from the server on the screen.

[1922] 2. Ensure that the information displayed on the device is in a visually easy-to-understand format.

[1923] Step 2: Notification of warnings

[1924] 1. The terminal notifies the user of warnings and instructions from the server audibly and visually.

[1925] 2. The device records whether the user acknowledged the warning and sends that information to the server.

[1926] Step 3: Receiving User Input

[1927] 1. The terminal receives additional information and judgment results entered by the user and transmits the data to the server in real time.

[1928] Step 4: Collecting emotion data

[1929] 1. The device uses an emotion engine to analyze the user's facial expressions, tone of voice, etc., and collects emotional data in real time.

[1930] Step 5: Record and archive data

[1931] 1. The device automatically records user operations and the countermeasures taken.

[1932] 2. The terminal periodically sends the recorded data to the server and stores it in a database.

[1933] User

[1934] Step 1: Check the data and assess the situation

[1935] 1. The user checks the information displayed on the device (such as abnormality notification and countermeasures).

[1936] 2. Based on the displayed information, the user decides on the best course of action to take, taking into account the actual driving situation and their own emotional state.

[1937] Step 2: Enter and execute instructions

[1938] 1. The user inputs the determined countermeasure into the terminal and executes the corresponding instructions.

[1939] 2. The user enters details of the countermeasures taken and the results into the terminal and sends the data to the server.

[1940] Step 3: Check for warnings and take action

[1941] 1. The user checks the warnings and instructions sent from the device and takes the necessary action promptly.

[1942] 2. The user records the details and circumstances of the response on the device and sends that information to the server.

[1943] Step 4: Regularly check and update

[1944] 1. The user regularly checks the monitoring data and warning information displayed on the device.

[1945] 2. Users should take preventative measures and preparations as necessary to prevent abnormalities from occurring.

[1946] Specific examples

[1947] Server operation example

[1948] For example, when monitoring for abnormalities caused by bad weather, the server collects past bad weather data and analyzes it using an AI model. It also collects data on the user's emotional state during bad weather and analyzes it using an emotion engine. If the server analyzes weather information in real time and detects an approaching storm, it determines the safety of a specific flight and sends the results to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[1949] Example of terminal operation

[1950] For example, when a device monitors a runway using a surveillance camera, it displays the video in real time. A warning of an intrusion into the runway from the server is sent to the user visually and audibly. At this time, the emotion engine analyzes the user's facial expression and tone of voice. When the user acknowledges the warning, the information is sent to the server, along with the analyzed emotion data.

[1951] Examples of user responses

[1952] For example, a user who has received information about an approaching storm may decide to delay or cancel their flight, or to choose an alternative route. In this case, the user may refer to the analysis results of the emotion engine to select the optimal response. The selected response is then entered into the device, and the information is sent to the server and shared with relevant parties immediately. Furthermore, the user's emotional state is also recorded and analyzed, and this information is used to improve the service.

[1953] This allows the system of the present invention to combine AI and sentiment analysis to further improve the safety and efficiency of air travel.

[1954] Example 2

[1955] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1956] Conventional systems for detecting anomalies and providing countermeasures in aviation operations are prone to human error and place a heavy psychological burden on users. In particular, in emergency situations, the user's emotional state is not properly taken into account, which can result in the selection of the most appropriate countermeasure. This has led to a demand for improved safety and efficiency.

[1957] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1958] In this invention, the server includes means for collecting past operation data and emotion data from a database, means for training an AI model for anomaly detection based on the collected data, means for receiving operation data in real time and detecting anomalies using the AI ​​model, means for generating countermeasures based on the detected anomalies and the user's emotional state, and means for transmitting the generated anomaly notification and countermeasures to the user terminal. This makes it possible to provide anomaly countermeasures that take the user's emotional state into consideration, thereby achieving improved safety and efficiency.

[1959] The "database" is an information collection point for storing and managing various data required by the system, such as past operation data and emotion data.

[1960] "Emotion data" is information that indicates the user's psychological state, and is data extracted from facial expressions, tone of voice, behavior, and the like.

[1961] An "AI model" is an algorithm that is trained based on collected data to detect anomalies and make predictions.

[1962] "Server" means a central control unit that collects, processes, and analyzes data and manages communications with user terminals.

[1963] A "user terminal" is an electronic device used by air traffic controllers and pilots to receive information and implement countermeasures.

[1964] "Anomaly detection" is the process of using AI models to analyze operational data and identify abnormal conditions or malfunctions.

[1965] "Countermeasures" refer to specific actions or instructions to be taken in response to detected abnormalities.

[1966] The "emotion engine" is software that analyzes the user's facial expressions and tone of voice, and collects and evaluates emotional data in real time.

[1967] An "abnormality notification" is a message or notification that notifies a user terminal of a detected abnormality and a countermeasure.

[1968] "Users" are human operators such as controllers and pilots who operate the system, review information, and implement countermeasures.

[1969] This invention combines an emotion engine with an aircraft operation support system to provide emergency response measures that take into account the user's emotional state. This system aims to reduce human error while also easing the user's psychological burden and improving safety and efficiency.

[1970] The system includes the following major components:

[1971] 1. Database

[1972] It stores past operation data, weather information, accident and incident history, user emotion data, etc. To do this, it uses a commonly used relational database (e.g., MySQL).

[1973] 2. AI Model

[1974] This is an algorithm trained on database information to perform anomaly detection and prediction. It uses the Python TensorFlow library. In particular, by including user emotional data, it is possible to provide emotion-based countermeasures for anomalies.

[1975] 3. Server

[1976] It collects, processes, and analyzes data. It receives operational data (flight information, aircraft status, weather information, etc.) in real time and provides it to an AI model to analyze operational status. The server receives real-time data using WebSocket technology and sends the analysis results to the user's device.

[1977] 4. User Device

[1978] This is an electronic device used by air traffic controllers and pilots to receive monitoring results and abnormality notifications. The terminal displays information sent from the server in real time and issues audio and visual warnings. It also sends additional information and judgment results entered by the user to the server in real time, and uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data.

[1979] 5. Emotion Engine

[1980] This is an engine that recognizes user emotions in real time and analyzes the data. The emotion engine uses a camera and microphone to analyze the user's facial expressions and tone of voice to collect emotional data. The collected data is sent to a server and used to generate countermeasures against abnormalities.

[1981] Specific examples

[1982] Server operation example

[1983] For example, when a server monitors for abnormalities caused by bad weather, it collects and learns from past bad weather data and analyzes it using an AI model. It also collects and learns data on the user's emotional state during past bad weather. If it analyzes weather information in real time and detects an approaching storm, it determines the safety of the flight and sends the result to the user's device. It also presents countermeasures prioritized according to the user's emotional state.

[1984] Example of terminal operation

[1985] For example, if a device monitors a runway using a surveillance camera, it will display the video in real time and notify the user visually and audibly of any runway intrusion warnings from the server. At this time, the emotion engine analyzes the user's facial expressions and tone of voice, and when the user acknowledges the warning, that information is sent to the server, which then presents prioritized countermeasures based on the detected anomaly.

[1986] Examples of user responses

[1987] For example, if a user sees information about an approaching storm, they can decide whether to delay or cancel their flight, or select an alternative route. In this case, they can select the optimal response, taking into account the analysis results of the emotion engine. The selected response is then entered into the device, and the information is sent to the server and immediately shared with relevant parties. Similarly, if a user receives an alert about a false runway intrusion, they can quickly implement a response and enter the results into the device. The emotion engine will then evaluate the user's emotional state, and the results will also be used to improve the service.

[1988] Prompt Sentence Examples

[1989] "Please explain how the server collects historical severe weather data and operational data to train the AI ​​model. Also, please describe the process by which the weather information is analyzed in real time and anomaly response measures are generated based on the user's emotional state."

[1990] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1991] Program processing flow

[1992] server

[1993] Step 1:

[1994] Data collection

[1995] The server connects to the flight database to retrieve flight records, weather information, past accident and incident data, and user sentiment data. These data are retrieved using a MySQL database.

[1996] Input: Database connection information

[1997] Processing: Extract the required data from the database using SQL queries.

[1998] Output: A set of operational and emotional data

[1999] Step 2:

[2000] Training an AI model

[2001] The server uses the Python TensorFlow library to train the AI ​​model, which learns patterns to detect anomalies based on collected operational and emotional data.

[2002] Input: A set of operational data and emotion data

[2003] Processing: After data preprocessing (normalization, feature extraction, etc.), training is performed using an AI model.

[2004] Output: A trained AI model

[2005] Step 3:

[2006] Receiving and analyzing real-time data

[2007] The server periodically receives real-time data (flight information, aircraft status, weather data, etc.) sent from the aircraft using WebSocket technology, and provides the received data to an AI model to analyze the current operating status.

[2008] Input: Real-time data

[2009] Processing: The received data is fed into an AI model for analysis to determine whether there are any anomalies.

[2010] Output: Current operating status and abnormality detection results

[2011] Step 4:

[2012] Detecting anomalies and generating countermeasures

[2013] The server uses an AI model to detect anomalies. If an anomaly is detected, it identifies the details (location of occurrence, scope of impact, cause, etc.), adjusts the priority of countermeasures based on data from the emotion engine, and then generates countermeasures.

[2014] Input: Anomaly detection results and emotion data

[2015] Action: Generate and prioritize countermeasures based on detected anomalies.

[2016] Output: Abnormal notification and countermeasures

[2017] Step 5:

[2018] Sending abnormality notifications and countermeasures

[2019] The server transmits the generated abnormality notification and countermeasure to the user terminal.

[2020] Input: Abnormal notification and countermeasures

[2021] Processing: Convert the notification data into an appropriate format and send it to the user terminal.

[2022] Output: Abnormal notification to user terminal

[2023] Terminal

[2024] Step 1:

[2025] Data Display

[2026] The terminal displays the monitoring results and abnormality notifications sent from the server in real time, using an interface that uses a GUI library.

[2027] Input: Abnormal notification and countermeasures

[2028] Processing: Updates various display elements of the GUI.

[2029] Output: Displayed abnormality notification and countermeasures

[2030] Step 2:

[2031] Warning Notification

[2032] The terminal will notify the user of warnings and instructions from the server audibly and visually, including playing audio files and visual alerts.

[2033] Input: Error notification and warning information

[2034] Action: Plays an audio and displays a visual alert.

[2035] Output: Audio and visual warnings

[2036] Step 3:

[2037] Receiving User Input

[2038] The terminal receives input from the user and transmits the data to the server in real time, either via a touchscreen or keyboard input.

[2039] Input: User-entered data

[2040] Processing: Sends the received user input to the server.

[2041] Output: Send input data to the server

[2042] Step 4:

[2043] Collecting Emotional Data

[2044] The device uses an emotion engine to analyze the user's facial expressions and tone of voice to collect emotional data, using a camera and microphone.

[2045] Input: User's facial expression data and voice data

[2046] Processing: Emotion data is generated using facial expression analysis algorithms and voice analysis algorithms.

[2047] Output: Emotion data

[2048] Step 5:

[2049] Data recording and archiving

[2050] The device automatically records the user's actions and the countermeasures taken, and transmits the data to the server using local storage and network communication.

[2051] Input: User operation log and execution details

[2052] Processing: The log data is stored in a format and sent to the server.

[2053] Output: Send log data to the server

[2054] User

[2055] Step 1:

[2056] Data confirmation and situation assessment

[2057] The user checks the information displayed on the device, determines the best course of action based on the operational status, and takes appropriate action based on the displayed information.

[2058] Input: Abnormality notification and countermeasures displayed on the terminal

[2059] Processing: Analyze the displayed information and make decisions based on the situation.

[2060] Output: Decision results and action selection

[2061] Step 2:

[2062] Entering and executing instructions

[2063] The user inputs the determined countermeasure into the terminal and executes the corresponding instruction.

[2064] Input: Workaround

[2065] Processing: Executes the input instructions and inputs the results to the terminal.

[2066] Output: Input data and execution results...

Claims

1. a means for collecting historical operational data from a database; A means to train an AI model for anomaly detection based on the collected data; and A means of receiving operational data in real time and detecting anomalies using an AI model; means for generating countermeasures for the detected anomalies; means for transmitting the generated abnormality notification and countermeasure to a user terminal; A system including:

2. A means for updating the anomaly detection result and the countermeasure based on the information input by the user; means for notifying the user terminal of the updated information again; The system of claim 1 further comprising:

3. A means for recording user operations and execution of countermeasures and storing the records in a database; A means for displaying the stored recorded data in a format that can be reviewed at a later time; The system of claim 1 further comprising:

Images