System

Abstract

A system is provided.SOLUTION: A system comprising: means for obtaining flight data; means for analyzing video and audio data; means for generating aircraft operation instructions based on an analysis result; means for detecting an emergency event; means for activating an autopilot system when the emergency event is detected; and means for continuously training a machine learning model using the flight data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The goals of the aviation industry are to improve safety and reduce the workload of pilots. Pilots face a variety of challenges, particularly during long flights and complex route planning. These challenges include mishearing or misinterpreting communications with air traffic controllers, responding immediately in emergencies, and understanding the situation in real time. These problems are caused by human error and overwork, and therefore have a high potential to compromise safety. The present invention aims to provide a system that solves these challenges. [Means for solving the problem]

[0005] The present invention provides a system including means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system upon detecting an emergency, and means for continuously training a machine learning model using the flight data. This system analyzes real-time data during flight and provides appropriate instructions to the pilot, thereby improving safety. Furthermore, by activating the autopilot system upon an emergency, the pilot's workload can be reduced. Furthermore, by continuously training the flight data, the accuracy of the system can be improved, further improving safety.

[0006] "Flight data" refers to real-time data such as altitude, speed, geographic coordinates, and engine status that indicate the aircraft's operational status.

[0007] "Video Data" means image or video data that visually records an airport or its surrounding environment.

[0008] "Audio data" refers to acoustic data such as the content of communications between air traffic controllers and pilots and sounds heard during flight.

[0009] "Means of analysis" refers to technology that uses artificial intelligence and machine learning models to analyze flight data, video data, and audio data and extract important information.

[0010] "Aircraft operation instructions" are specific operation instructions and recommended actions for the pilot that are generated based on the analysis results.

[0011] An "emergency" refers to an abnormal or dangerous situation that occurs during flight and requires immediate action.

[0012] An "autopilot system" is an automated control system that enables an aircraft to navigate autonomously and respond to emergencies.

[0013] A "machine learning model" is an algorithm that learns from past data and analyzes and predicts new data.

[0014] "Continuous learning" is a technique that uses real-time and historical flight data to improve the performance of machine learning models. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] ---

[0037] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0038] System Configuration

[0039] 1. Real-time data acquisition

[0040] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0041] 2. Analysis of video and audio data

[0042] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0043] 3. Data analysis using AI models

[0044] The server uses an AI model (machine learning algorithm) to analyze the flight data and analyzed video and audio data, thereby extracting risk factors related to flight safety and supporting appropriate decisions based on the analysis results.

[0045] 4. Providing instructions to pilots

[0046] The server then uses the analysis to provide instructions to the pilot in text and audio format, such as "maintain altitude" and "follow the designated route."

[0047] 5. Emergency Detection

[0048] The server uses the analyzed data to detect abnormalities and emergencies in real time, including engine malfunctions, sudden weather changes, and close encounters with other aircraft.

[0049] 6. Activating the autopilot system

[0050] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0051] 7. Continuous learning

[0052] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0053] ---

[0054] Processing flow and specific examples

[0055] Real-time data acquisition

[0056] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0057] Video and audio data analysis

[0058] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0059] Data analysis using AI models

[0060] The server inputs all acquired data into an AI model and extracts safety-related risk factors. For example, the analysis results show that sudden weather changes are predicted.

[0061] Providing instructions to pilots

[0062] Based on the analysis results, the server provides instructions to the pilot via text and voice notification, such as "Please change to a new route" to respond to weather changes.

[0063] Emergency Detection

[0064] The server analyzes flight data and detects emergencies such as engine malfunctions or sudden weather changes. For example, if an abnormally high engine temperature is detected, a notification will be sent.

[0065] Autopilot system activation

[0066] If an emergency is detected, the server will activate the autopilot system and take appropriate action, such as specifying an emergency landing route to an appropriate airport in the event of an engine malfunction.

[0067] Continuous learning

[0068] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns.

[0069] ---

[0070] As described above, the system of the present invention can improve aircraft safety and reduce the burden on pilots. This system utilizes advanced data analysis technology and machine learning to set a new standard in the aviation industry.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The server receives real-time flight data from various sensors on the plane. This data includes altitude, speed, geographic coordinates, and engine status. For example, if the plane is at an altitude of 10,000 feet and traveling at 550 knots, that data is sent to the server.

[0074] Step 2:

[0075] The server acquires video data from the airport. It collects real-time camera footage from within the airport and uses image recognition technology to analyze the runway conditions and the presence of obstacles in the surrounding area. For example, it can detect foreign objects on the runway from the camera footage.

[0076] Step 3:

[0077] The server acquires the voice data of the communication between the controller and the pilot and analyzes it using voice recognition technology. This detects erroneous instructions and mishearing by the pilot. For example, if the controller instructs "Turn right" but the pilot mistakenly interprets it as "Turn left," the system will detect the misunderstanding.

[0078] Step 4:

[0079] The server inputs the flight data, video data, and audio data it acquires into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors related to flight safety. For example, if a sudden change in weather is predicted, that information is extracted as a risk factor.

[0080] Step 5:

[0081] Based on the analysis results, the server generates appropriate instructions for the pilot. The generated instructions are displayed in text format on the cockpit display and also communicated to the pilot in voice format. For example, the instruction "Maintain altitude" is displayed and the voice also communicates "Maintain altitude."

[0082] Step 6:

[0083] The server detects emergencies from the analysis data, such as engine abnormalities, sudden weather changes, or close proximity to other aircraft. For example, if the engine temperature is abnormally high, it will be detected as an emergency.

[0084] Step 7:

[0085] When an emergency is detected, the server activates the autopilot system, which then executes the appropriate emergency response procedures. For example, in the event of an engine failure, the autopilot system calculates an emergency landing route to the nearest airport and autonomously directs the plane along that route.

[0086] Step 8:

[0087] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, new flight data can be used to refine the fuel efficiency optimization algorithm for the next flight.

[0088] These are the processing steps of the system of the present invention, including specific examples, which improves aircraft safety and reduces pilot workload.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Improving safety and reducing pilot workload are key issues in modern aviation. In particular, failure to properly detect anomalies and respond to emergencies in real time poses a risk of serious accidents. Systems that efficiently analyze flight data and provide accurate flight instructions to pilots are also needed. Furthermore, continuous system performance improvement through machine learning is also necessary.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes a means for acquiring flight data, a means for analyzing surveillance camera and wireless communication data, a means for generating aircraft operation instructions based on the analysis results, a means for detecting an emergency, a means for activating an autopilot system when the emergency is detected, and a means for continuously training a machine learning model using the flight data and analysis data. This enables abnormalities to be detected in real time and appropriate countermeasures to be implemented promptly. Furthermore, continuous learning improves the accuracy of the system, contributing to aircraft safety and reducing the burden on pilots.

[0094] "Flight data" is data that includes information such as aircraft altitude, speed, geographic coordinates, and engine status.

[0095] A "surveillance camera" is a device that captures images for the purpose of monitoring airports and aircraft.

[0096] "Radio communication data" refers to data of voice communications between air traffic controllers and pilots.

[0097] The "analysis results" are the analysis results obtained after analyzing the acquired data.

[0098] "Aircraft operation instructions" refers to specific instructions regarding the piloting and operation of an aircraft based on the analysis results.

[0099] An "emergency situation" is an abnormal condition that poses a significant risk to the safe operation of an aircraft.

[0100] An "autopilot system" is a system for automatically operating an aircraft.

[0101] A "machine learning model" refers to an algorithm or system that automatically learns from data and makes predictions and classifications.

[0102] "Continuous learning" is the process of constantly updating a model's learning with new data to improve its accuracy and performance.

[0103] MODE FOR CARRYING OUT THE INVENTION

[0104] The following describes in detail the mode for carrying out the present invention. The purpose of this system is to improve aircraft safety and reduce the burden on pilots. The hardware and software used, as well as the methods of data processing and calculation, are explained below.

[0105] System Configuration

[0106] Real-time data acquisition

[0107] The server obtains flight data in real time from various sensors on the aircraft. This flight data includes altitude, speed, geographic coordinates, and engine status. The server collects data using the aircraft's altimeter, speedometer, GPS, and engine monitoring system, and stores it in a database (e.g., MySQL) in real time.

[0108] As a specific example, the server obtains information in seconds about an airplane in flight, such as its altitude, speed, latitude, and longitude, which are 3,000 meters, 900 km / h, 35 degrees north, and 139 degrees east, and stores this information in a database.

[0109] Video and audio data analysis

[0110] The server collects and analyzes airport surveillance camera footage and radio communication data between controllers and pilots. Specifically, the server acquires surveillance camera footage and uses image recognition software (e.g., OpenCV) to extract important visual information. It also analyzes controller-pilot communications using voice recognition technology (e.g., Google Speech-to-Text API) to detect miscommunications and mishearing.

[0111] Specifically, the server detects moving objects on the runway from surveillance camera footage and uses voice recognition to confirm that the controller's instruction is to "climb 500 feet."

[0112] Data analysis using AI models

[0113] The server inputs the acquired data into a machine learning model for analysis. The server inputs flight data and video / audio analysis data acquired from the database into an AI model (e.g., TensorFlow), analyzes the data, and extracts potential risk factors. The analysis results are classified by risk type and sent to the server's internal evaluation system.

[0114] As a specific example, the server obtains predictions of sudden weather changes from an AI model and reflects this information in the risk assessment system.

[0115] Providing instructions to pilots

[0116] The server provides text and voice instructions to the pilot based on the analysis results. For example, the server checks the analysis results, generates text instructions, and displays them on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. A specific example of such an instruction would be "Please change to a new route" displayed on the screen and also given as a voice instruction.

[0117] Emergency Detection

[0118] The server detects emergencies based on real-time data analysis results. The server periodically monitors the analysis results and runs algorithms to detect abnormal patterns. If an emergency is detected, it immediately generates an alert and notifies relevant parties. Detailed information about the emergency is provided to the pilot.

[0119] As a specific example, the server detects an abnormal rise in engine temperature and notifies the pilot with an alert on the screen and via voice.

[0120] Autopilot system activation

[0121] When the server detects an emergency, it activates the autopilot system. When the server detects an emergency, it issues a command to activate the autopilot mode. It also sends a pre-defined emergency response route to the autopilot system and executes it. The autopilot system then initiates the designated emergency response.

[0122] As a specific example, if an engine malfunction is detected, the server will activate the autopilot system and route the aircraft to the nearest suitable airport for an emergency landing.

[0123] Continuous learning

[0124] The server continuously trains the AI ​​model using the acquired data, adds newly acquired flight data to the machine learning dataset, and periodically retrains the machine learning model to improve accuracy and performance, and deploys the updated model to the system.

[0125] For example, new flight data can be used to retrain AI models, improving the accuracy of anomaly detection.

[0126] Prompt Sentence Examples

[0127] "Please explain the specific processing flow of the system that acquires real-time data from the aircraft, analyzes video and audio to detect abnormalities, and issues instructions to the pilot. Please also explain the specific operation of each step."

[0128] As described above, the system of the present invention aims to improve aircraft safety and reduce pilot workload by utilizing advanced data analysis technology and machine learning, and this system can set a new standard in the aviation industry.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Program processing flow

[0131] Step 1:

[0132] The server acquires flight data in real time from various sensors on the aircraft. The input for this step is data from the altimeter, speedometer, GPS, and engine monitoring system installed on the aircraft. The server receives this data and stores it in a database (e.g., MySQL) in real time. Specifically, the server acquires data every second, such as an altitude of 3,000 meters, a speed of 900 km / h, latitude 35 degrees north, and longitude 139 degrees east, and stores it in the database.

[0133] Step 2:

[0134] The server collects and analyzes airport surveillance camera footage and radio communication data between the controller and pilot. The input for this step is live video data from the surveillance cameras and audio data from radio communication. The server uses OpenCV to analyze the video data and detect moving objects and anomalies. It also uses the Google Speech-to-Text API to convert audio data into text and detect erroneous instructions or mishearing. Specifically, the server detects foreign objects on the runway from surveillance camera footage and uses voice recognition to confirm the controller's instruction to "climb to an altitude of 500 feet."

[0135] Step 3:

[0136] The server inputs the acquired flight data and analyzed video and audio data into the AI ​​model for analysis. The inputs for this step are the flight data, video data, and audio data acquired in the previous step. The server uses TensorFlow to analyze this data and extract potential risk factors. The analysis results are classified by risk type and sent to the built-in evaluation system. Specifically, the server receives a result from the AI ​​model indicating that a sudden weather change is predicted, and reflects this information in the risk evaluation system.

[0137] Step 4:

[0138] Based on the analysis results, the server provides text and voice instructions to the pilot. The input for this step is the AI ​​analysis results. The server checks the analysis results, generates instructions, and displays them in text on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. Specifically, the instruction "Please change to a new route" is displayed on the screen and is also given voice instructions.

[0139] Step 5:

[0140] The server detects emergencies based on the results of real-time data analysis. The input to this step is the analysis results. The server periodically monitors the analysis results and runs an algorithm to detect abnormal patterns. If an emergency is detected, an alert is immediately generated and notified to the pilot. Specifically, the server detects an abnormal rise in engine temperature and notifies the pilot of the alert on the screen and by voice.

[0141] Step 6:

[0142] If the server detects an emergency, it activates the autopilot system. The input to this step is the emergency detection result. The server issues a command to activate the autopilot mode and sends a pre-set emergency response route to the autopilot system to execute. Specifically, if an engine abnormality is detected, the autopilot system is activated and an emergency landing route to the nearest appropriate airport is specified.

[0143] Step 7:

[0144] The server continuously trains the AI ​​model using the acquired data. The input for this step is newly acquired flight data. The server adds this to the machine learning dataset and periodically retrains the machine learning model to improve accuracy and performance. The updated model is then deployed to the system. Specifically, the AI ​​model is retrained using new flight data, improving the accuracy of anomaly detection.

[0145] Through the above processing steps, this system can improve aircraft safety and reduce the burden on pilots.

[0146] (Application example 1)

[0147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] Operations at conventional logistics centers depended heavily on human labor, posing safety and efficiency challenges. For example, when equipment failure, abnormal temperature rises, or operational errors occurred, it was difficult to immediately detect and respond to these risks. Furthermore, due to insufficient real-time data analysis and automation, there were frequent delays in response in situations where appropriate decisions needed to be made immediately. Therefore, there was an urgent need to provide a system that would improve the safety and efficiency of logistics centers.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0150] In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data, means for acquiring various data within the logistics center in real time, means for providing instructions to workers and automated transport robots based on the analysis results, and means for automatically executing an emergency response in the event of an abnormality. This makes it possible to analyze real-time data from sensors and surveillance cameras in the logistics center and automate the provision of instructions to workers and automated transport robots and emergency responses.

[0151] "Flight data" refers to various data relating to the operation of an aircraft, including the aircraft's altitude, speed, geographic coordinates, engine status, etc.

[0152] "Video Data" means images or video information captured by surveillance cameras or other visual sensors.

[0153] "Voice data" means data containing communications between air traffic controllers and pilots and other voice information.

[0154] "Analysis results" refer to the results of analyzing the acquired data using AI models or machine learning algorithms.

[0155] "Aircraft operation instructions" refer to specific instructions regarding aircraft operation based on the analysis results, such as instructions to maintain altitude or change route.

[0156] An "emergency" is an abnormal condition, such as an engine malfunction or a sudden change in weather, that poses a significant risk to aircraft operation.

[0157] An "autopilot system" is an automated flight system that assists or substitutes for pilot operations under certain conditions, such as emergencies.

[0158] A "machine learning model" is an algorithm or program that continuously learns from collected data and improves the accuracy of analysis.

[0159] A "logistics center" is a facility that receives, stores, and ships goods.

[0160] "Real-time data" refers to data that is obtained instantly from sensors, surveillance cameras, etc. and processed without delay.

[0161] "Workers" refers to workers who work within a logistics center.

[0162] An "automatic transport robot" is a robot that performs automated transport tasks within a logistics center.

[0163] "Providing instructions" refers to the act of communicating the work content and response methods to workers and automatic transport robots based on the analysis results.

[0164] An "abnormal condition" is a deviation from normal operating conditions that may affect safety or efficiency.

[0165] "Emergency response" refers to immediate response measures taken in the event of an abnormality or emergency.

[0166] The following describes in detail a system for implementing an application example of the present invention.

[0167] System Configuration

[0168] This system is designed to support efficient and safe operations in logistics centers and primarily consists of a server, sensors, surveillance cameras, terminals (smartphones or tablets) used by workers, and an automated transport robot.

[0169] Hardware and Software Use

[0170] Hardware:

[0171] Various sensors (temperature, humidity, location information, etc.)

[0172] surveillance cameras

[0173] Smartphone or tablet

[0174] Automatic transport robot

[0175] software:

[0176] Python for Data Analysis (Flask Framework)

[0177] Machine learning models (scikit-learn and TensorFlow)

[0178] WebSocket (JavaScript) for real-time data acquisition

[0179] Data processing and calculation

[0180] The server receives real-time data from various sensors in the logistics center via WebSocket. The received data is analyzed using a Flask-based API. A pre-trained machine learning model (scikit-learn or TensorFlow) is used for the analysis, and this model detects risk factors such as temperature rises or abnormal machine operation.

[0181] Operations and Examples

[0182] Operation steps:

[0183] 1. Real-time data acquisition: Collect data such as temperature, humidity, and location information from sensors within the logistics center.

[0184] 2. Sending data: The collected data is sent to the server via WebSocket.

[0185] 3. Data analysis: The server's machine learning model analyzes the data and extracts risk factors.

[0186] 4. Providing instructions: Based on the analysis results, appropriate instructions are provided in voice and text format to the worker's device and the automatic transport robot.

[0187] 5. Response to abnormalities: If an abnormality is detected, an emergency response will be immediately implemented and the system will take autopilot.

[0188] Examples:

[0189] For example, if a temperature sensor detects an abnormal temperature rise in an area of ​​a logistics center, this data is sent to a server in real time. The server's machine learning model uses this information to determine that there is a high risk of fire. The system immediately provides a voice instruction to workers saying, "Abnormal temperature rise detected in Area B. Please check immediately," and instructs an automated transport robot to move from the area to a safe location.

[0190] Example prompt sentence:

[0191] Design an AI system to collect and analyze real-time data from various sensors in a logistics center. Based on the analysis results, provide instructions to workers and automated transport robots, and automatically respond in the event of an abnormality.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] The server collects data in real time from various sensors in the logistics center. As input, it receives data such as temperature, humidity, and location information from each sensor, and as output, it stores this data in its internal storage.

[0195] Specifically, the system obtains the value measured by the temperature sensor (e.g., 25 degrees) and the coordinates obtained by the position sensor (e.g., 35 degrees north latitude, 139 degrees east longitude).

[0196] Step 2:

[0197] The server collects video data from the surveillance cameras and audio data from the microphones. As input, it receives video frames from the cameras and audio data from the microphones, and as output, it passes these data to the video analysis and audio analysis modules.

[0198] Specifically, it captures camera video frames (e.g., JPEG images with a resolution of 1920x1080) and records audio data (e.g., conversations between controllers and workers).

[0199] Step 3:

[0200] The server inputs the collected data into a machine learning model for analysis. Data from sensors and surveillance cameras is received as input, and risk factors are extracted as analysis results as output. Specific data processing, for example, involves analyzing temperature data over time to detect abnormal temperature increases.

[0201] Specific actions include identifying abnormal patterns (e.g., a temperature spike lasting 10 minutes) and reporting them as a risk.

[0202] Step 4:

[0203] The server provides instructions to workers and automated transport robots based on the analysis results. It receives the risk factors of the analysis results as input, generates instructions in text and voice format as output, and sends them to the workers' terminals and the robots.

[0204] Specifically, the system sends a voice message to the worker's smartphone saying, "An abnormal temperature rise has been detected in Area B. Please check immediately," and instructs the robot to move to a safe location.

[0205] Step 5:

[0206] The server automatically executes emergency response when an anomaly is detected. It receives anomaly detection alerts as input and executes specific response measures as output.

[0207] Specifically, if it is determined that there is a high risk of fire, the system will immediately contact the fire department and move the automated transport robot to a safe location.

[0208] Step 6:

[0209] The server continuously trains the machine learning model using data collected in real time, taking newly collected data as input and improving the model's performance as output.

[0210] Specifically, the system uses previously collected data on normal and abnormal patterns as training data and updates the model to improve analysis accuracy.

[0211] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0212] ---

[0213] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0214] System Configuration

[0215] 1. Real-time data acquisition

[0216] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0217] 2. Analysis of video and audio data

[0218] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0219] 3. User Emotion Recognition

[0220] The server provides the pilot's voice and biological information to the emotion engine, which analyzes the user's (pilot's) emotions. The emotion engine recognizes the pilot's emotional state from voice tone and physiological data (heart rate, skin potential response, etc.).

[0221] 4. Data analysis using AI models

[0222] The server inputs the flight data, video data, audio data, and emotion data into the AI ​​model for analysis. The AI ​​model integrates this data and extracts risk factors related to flight safety.

[0223] 5. Providing instructions to pilots

[0224] The server then uses the analysis results to provide instructions to the pilot in text and voice format. Using the results of the emotion engine, the server generates appropriate instructions based on the pilot's emotional state. For example, if the pilot is under high stress, the server provides concise and clear instructions, adding reassuring words.

[0225] 6. Emergency Detection

[0226] The server uses the analysis data to detect abnormal situations and emergencies in real time. It also takes into account data from the emotion engine to more accurately detect emergencies. For example, if a pilot's heart rate suddenly rises or their emotional state suddenly changes, it will notify them of that condition as a warning.

[0227] 7. Activating the autopilot system

[0228] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0229] 8. Continuous learning

[0230] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0231] ---

[0232] Processing flow and specific examples

[0233] Real-time data acquisition

[0234] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0235] Video and audio data analysis

[0236] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0237] User Emotion Recognition

[0238] The server analyzes the pilot's emotional state using an emotion engine. Based on collected physiological data such as voice tone and heart rate, it determines whether the pilot is under stress. For example, if the heart rate is abnormally high and the voice tone is tense, it determines that the pilot is under high stress.

[0239] Data analysis using AI models

[0240] The server inputs all acquired data into an AI model to extract safety-related risk factors. For example, the analysis results show that a sudden change in weather is predicted.

[0241] Providing instructions to pilots

[0242] Based on the analysis results, the server provides instructions to the pilot via text display and voice notification. The emotion engine's analysis results are reflected in the instructions generated based on the pilot's emotional state. For example, in a high-stress situation, additional instructions such as "Remain calm and maintain your current altitude" are given.

[0243] Emergency Detection

[0244] The server analyzes flight data to detect emergencies such as engine malfunctions or sudden weather changes. It also takes into account data from the emotion engine, and if an abnormal physiological state of the pilot is observed, it uses that information to issue a highly accurate warning. For example, if the engine temperature is abnormally high and the pilot's heart rate is also rising sharply, this will be detected as an emergency.

[0245] Autopilot system activation

[0246] If the server detects an emergency, it immediately activates the autopilot system and instructs the appropriate emergency response. For example, in the event of an engine malfunction, it will instruct the aircraft to make an emergency landing at an appropriate airport, and the autopilot system will fly that route autonomously.

[0247] Continuous learning

[0248] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns, for example, by improving the system's response algorithms based on newly observed emergency patterns.

[0249] The above are the processing steps of the system of the present invention, including specific examples. By combining it with an emotion engine, the safety of aircraft can be further improved and the mental burden on pilots can be effectively reduced.

[0250] The processing flow will be explained below.

[0251] Step 1:

[0252] The server receives real-time flight data from various sensors on the aircraft, including altitude, speed, geographic coordinates, and engine status. For example, the flight altitude is 30,000 feet, the speed is 500 knots, the geographic coordinates are 35 degrees north, 140 degrees east, and the engine status is normal.

[0253] Step 2:

[0254] The server receives camera footage from the airport. The video data is analyzed using image recognition technology to detect obstacles and abnormalities on the runway. For example, if a bird is detected on the runway through video analysis, it will identify it and issue a warning.

[0255] Step 3:

[0256] The server receives voice data from radio communications between the controller and the pilot and analyzes it using voice recognition technology. For example, it checks whether the controller's instructions, such as "Maintain current altitude," were properly conveyed to the pilot.

[0257] Step 4:

[0258] The server provides the pilot's voice, heart rate, and other physiological information to the emotion engine, which analyzes the user's (pilot's) emotional state. The emotion engine recognizes the pilot's stress and tension from the tone of their voice and heart rate. For example, if their heart rate is higher than normal and their voice tone is rising, it determines that the pilot is nervous.

[0259] Step 5:

[0260] The server inputs flight data, video data, audio data, and emotion data into the AI ​​model, and analyzes risk factors related to flight safety. For example, sudden weather changes and close proximity to other aircraft are extracted as risk factors.

[0261] Step 6:

[0262] The server generates appropriate instructions for the pilot based on the analysis results. It also reflects the results of the emotion engine and generates instructions that correspond to the pilot's emotional state. For example, it may say, "Please relax. You are instructed to maintain your current altitude."

[0263] Step 7:

[0264] The server detects emergencies from the analyzed data, including engine abnormalities, sudden weather changes, close encounters with other aircraft, etc. For example, if the engine temperature rises sharply and the pilot's heart rate also increases, this will be detected as an emergency.

[0265] Step 8:

[0266] If an emergency is detected, the server immediately activates the autopilot system and instructs the necessary emergency response. For example, in the event of an engine malfunction, the autopilot system can set up an emergency landing route to the nearest safe airport and fly that route autonomously.

[0267] Step 9:

[0268] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, it can learn new emergency patterns and improve prediction accuracy for the next flight.

[0269] The above are the detailed processing steps of the system of the present invention that combines the emotion engine. This is expected to further improve aircraft safety and effectively reduce the mental burden on pilots.

[0270] Example 2

[0271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Current aircraft systems face the problem of not being able to ensure flight safety or sufficiently reduce the pilot's mental burden by simply analyzing flight, video, and audio information. Failure to detect emergencies and provide appropriate instructions to pilots in real time increases the risk of serious accidents. Furthermore, there is no way to provide instructions that take into account the pilot's emotional state and stress level, which does not reduce the pilot's mental burden. A new system is needed to solve these issues and achieve safer and more efficient flight.

[0273] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0274] In this invention, the server includes means for acquiring flight information, means for analyzing video and audio information, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight information, means for analyzing the emotional state of the pilot from physiological information and audio information, and means for generating appropriate aircraft operation instructions using the emotional state analysis results, thereby improving flight safety and reducing the mental burden on the pilot.

[0275] "Flight information" refers to real-time data such as an aircraft's altitude, speed, geographic coordinates, and engine status.

[0276] "Video and audio information" refers to digital video and audio data obtained from airport surveillance camera footage and radio communications with air traffic controllers.

[0277] "Analysis results" refers to the data obtained by processing flight information, video, and audio information, and specifically includes the detection of erroneous instructions and the extraction of risk factors.

[0278] "Aircraft operation instructions" refer to operational guidance and action instructions for the pilot that are generated based on the analysis results.

[0279] An "emergency" refers to a situation that threatens the safety of an aircraft, such as an engine malfunction, a sudden change in weather, or a physiological abnormality in the pilot.

[0280] An "autopilot system" is a system that operates an aircraft autonomously without pilot intervention in the event of an emergency.

[0281] "Machine learning model" refers to a statistical model that uses algorithms to accumulate knowledge and continuously learn, and includes models that make predictions and decisions based on flight information.

[0282] "Physiological information" refers to data that indicates the pilot's physical condition, such as their heart rate and skin potential response.

[0283] "Emotional state" refers to the pilot's psychological state as analyzed from their vocal tone and physiological information.

[0284] "Emotional state analysis results" refers to data on the pilot's emotional state analyzed by the emotion engine, including stress levels and tension.

[0285] "Appropriate aircraft operation instructions" refers to operational guidance and behavioral instructions that correspond to the pilot's mental condition and are generated based on the results of emotional state analysis.

[0286] MODE FOR CARRYING OUT THE INVENTION

[0287] An embodiment of the system of the present invention is described in detail below.

[0288] This system aims to improve aircraft safety by collecting and analyzing aircraft flight data, video and audio data, and physiological information. In particular, it analyzes the pilot's emotional state and generates appropriate instructions, thereby reducing the pilot's mental burden.

[0289] The server receives real-time flight data from the plane's various sensors. This includes the plane's altitude, speed, geographic coordinates, and engine status. For example, altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east. This data is retrieved from the aircraft's central system using a dedicated protocol.

[0290] The server collects video and audio data from airport surveillance cameras and wireless communications with air traffic controllers. Video data is acquired as streaming from the surveillance cameras, and audio data is received in digital audio format. Specifically, the system uses video analysis algorithms to detect foreign objects on the runway and voice recognition technology to analyze air traffic controllers' instructions in real time to check for erroneous instructions.

[0291] The server provides the pilot's voice and biometric information to the emotion engine, which analyzes their emotional state. The emotion engine analyzes their voice tone, heart rate, skin potential response, and other factors to recognize their emotional state. Data is collected from the audio microphone and biometric sensors and processed in real time. For example, if the pilot's heart rate is 30% higher than normal and their voice tone sounds tense, it is determined that the pilot is experiencing high levels of stress.

[0292] All data collected by the server (flight data, video data, audio data, emotion data) is input into the AI ​​model for analysis. The AI ​​model is composed of multiple neural networks and integrates and analyzes data. For example, sudden changes in weather are extracted as analysis results and reported as risk factors.

[0293] Based on the analysis results, the server provides instructions to the pilot in text and voice format. The instructions are generated based on the results of the emotion engine. For example, a pilot in a state of high stress may be given voice instructions such as "Remain calm and maintain your current altitude," which are also displayed on the screen in text format.

[0294] The server analyzes flight data in real time to detect abnormalities and emergencies. Emergencies include engine malfunctions, sudden weather changes, and abnormal pilot physiological states. For example, if the engine temperature suddenly rises and the pilot's heart rate also rises sharply, this will be detected as an emergency and an alert will be sent immediately.

[0295] When the server detects an emergency, it immediately activates the autopilot system, which executes emergency responses based on pre-programmed scenarios. For example, if an engine malfunction is detected, the autopilot system will execute an emergency landing route to the nearest airport.

[0296] The server continuously collects flight data and periodically updates the AI ​​model. This update process is performed offline, and new data sets are used to improve the model's performance. For example, new emergency scenarios can be learned and the system can adapt to them.

[0297] Prompt Sentence Examples

[0298] "The pilot's heart rate has been consistently 30% higher than normal. Please provide a specific example of what instructions should be given to the pilot if this condition is detected."

[0299] The above is an embodiment of the present invention, which can improve the safety of aircraft and effectively reduce the mental burden on pilots.

[0300] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0301] Step 1:

[0302] The server obtains flight data in real time from the aircraft's sensors. Specifically, data such as the aircraft's altitude, speed, geographic coordinates, and engine status are obtained from the aircraft's central system using a dedicated protocol. The input data is raw data from the sensors, which is collected in real time and stored in a database. As an output, various flight data are stored in the database in an organized manner.

[0303] Step 2:

[0304] The server acquires video data from surveillance cameras within the airport and collects audio data from radio communications with air traffic controllers. The video data is received in streaming format, and the audio data is received in digital audio format. The video data is run through an image recognition algorithm to detect foreign objects on the runway, and the audio data is run through voice recognition technology to check for erroneous instructions. The output is compiled into a report containing the foreign object detection results and audio analysis results.

[0305] Step 3:

[0306] The server collects the pilot's voice and physiological information and provides it to the emotion engine. Voice is captured through a microphone, and physiological data is acquired through a heart rate sensor and skin potential sensor. The emotion engine analyzes this data and recognizes the pilot's emotional state. The input is the voice and physiological data collected from the pilot, and the output is the pilot's emotional state (e.g., high stress, relaxed).

[0307] Step 4:

[0308] The flight data, video data, audio data, and emotional data acquired by the server are input into the AI ​​model. The AI ​​model includes a neural network that integrates and analyzes this data. The input data is initial data from various sensors, and the output is a report containing the integrated analysis results (e.g., sudden weather changes and risk factors). Specifically, the model performs functions such as detecting anomalies in flight data, detecting foreign objects in video data, and detecting erroneous instructions in audio data.

[0309] Step 5:

[0310] Based on the analysis results, the server provides instructions to the pilot in text and voice format. Instructions are generated based on the results of the emotion engine, and the pilot is given appropriate operational guidance. Analysis results and emotion data are used as input data, and the output is voice instructions (e.g., "Please remain calm and maintain your current altitude") and text instructions displayed on the pilot's display.

[0311] Step 6:

[0312] The server analyzes flight data and emotional data in real time to detect emergencies. If an engine abnormality, a sudden change in weather, or a pilot's physiological abnormality is detected, an emergency alert is issued immediately. The input is real-time flight data and emotional data, and the output is a warning message about the emergency and detailed information.

[0313] Step 7:

[0314] When the server detects an emergency, it activates the autopilot system. The autopilot system then takes appropriate action based on pre-programmed emergency response scenarios. For example, if an engine malfunction is detected, an emergency landing route to the nearest airport is automatically set and the flight is carried out along that route. The input is the emergency detection information, and the output is the activation of the autopilot system and a log of its actions.

[0315] Step 8:

[0316] The server continuously accumulates collected flight data and uses it to improve the performance of the AI ​​model. The newly acquired data is used to retrain the model, enabling more accurate analysis. The input is continuously collected flight data, and the output is an updated AI model and its performance evaluation results. For example, it is possible to learn new patterns of abnormal situations and incorporate appropriate countermeasures into the system.

[0317] (Application example 2)

[0318] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] In modern manufacturing, improving the efficiency and safety of factory operations are important challenges. In particular, there is a need to monitor the work environment and worker status in real time and respond quickly and appropriately when an abnormality occurs. However, existing systems lack the technology to integrate and analyze multiple data sources and automatically generate work instructions, leaving room for improvement. Therefore, there is a need for a system that can efficiently integrate and analyze real-time data and provide work instructions, including emergency responses.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data and environmental data, means for acquiring sensor data within the factory, means for analyzing environmental data and physiological data, means for generating work instructions based on the analysis results, means for autonomously responding to emergencies based on the results of the analysis means, and means for updating the generating AI model and optimizing the instruction provision. This enables efficient work management and improved safety within the factory.

[0321] "Flight data" is data including aircraft altitude, speed, geographic coordinates, and engine status.

[0322] "Video and audio data" refers to data that includes video and audio from a surveillance camera.

[0323] "Aircraft operation instructions" are instructions regarding flight operations sent from the server to the pilot.

[0324] "Emergency" means an abnormal or dangerous condition in aircraft operations or factory operations.

[0325] An "autopilot system" is a system that controls the automatic operation of an aircraft.

[0326] A "machine learning model" is an artificial intelligence model that is trained to perform analysis and predictions using collected data.

[0327] "Sensor data" is data obtained from various sensors within the factory (temperature, humidity, machine operating status, etc.).

[0328] "Environmental data" refers to data including environmental conditions such as temperature, humidity, noise, and vibration within a factory.

[0329] "Physiological data" refers to data that indicates the physiological state of a worker, such as heart rate and skin potential.

[0330] "Work instructions" are specific instructions for actions provided to workers based on the analysis results.

[0331] A "generative AI model" is an artificial intelligence model that generates new instructions and countermeasures by analyzing data.

[0332] "Autonomous emergency response" means that the system automatically takes action to deal with an emergency.

[0333] As an embodiment of the present invention, the system is described in detail below. This system collects and analyzes flight data, video and audio data, sensor data, environmental data, physiological data, etc. in real time, and issues appropriate instructions and emergency responses based on the analysis results.

[0334] System Configuration

[0335] 1. Real-time data acquisition

[0336] The server receives real-time flight data from various sensors on the plane, including altitude, speed, geographic coordinates, and engine status, as well as sensor data from the factory, such as temperature, humidity, and machine operation status.

[0337] 2. Analysis of video and audio data

[0338] The server acquires surveillance camera footage and environmental audio, and uses video recognition technology to extract important visual information and audio recognition technology to detect signs of abnormalities.

[0339] 3. Physiological Data Collection and Analysis

[0340] The server collects physiological data such as the worker's heart rate and provides it to the emotion engine to analyze the worker's stress level. This analysis determines the worker's workload and takes the necessary measures.

[0341] 4. Data analysis using AI models

[0342] The server inputs the acquired flight data, video and audio data, sensor data, and physiological data into the AI ​​model, which then integrates and analyzes this data to identify risk factors both inside and outside the factory, contributing to improved safety and efficiency.

[0343] 5. Instruction Generation and Delivery

[0344] Based on the analysis, the server generates appropriate instructions for an aircraft pilot or a factory worker, delivered in text or audio format and including specific advice based on the worker's emotional state.

[0345] 6. Emergency Detection and Autonomous Response

[0346] The server uses the analysis data to detect abnormalities and emergencies in real time, and activates the autopilot system or the factory's emergency response system as necessary, allowing for a quick and safe response.

[0347] 7. Continuous learning

[0348] The server continuously updates the machine learning model using the collected data to improve the performance of the entire system, allowing it to quickly incorporate new abnormal patterns.

[0349] Hardware and software used

[0350] The hardware used is as follows:

[0351] Sensors: Various sensors that collect environmental data within the factory in real time

[0352] Camera: A surveillance camera for capturing video data

[0353] Physiological data monitor: A monitor that acquires information such as the worker's heart rate

[0354] The software used is as follows:

[0355] Pandas: Real-time data acquisition in dataframe format

[0356] OpenCV: Camera image analysis

[0357] Librosa: Analysis of audio data

[0358] Scikit-learn: Building and analyzing machine learning models

[0359] Specific examples

[0360] For example, if a factory machine detects abnormal vibrations, the system will immediately analyze the vibration data and provide appropriate instructions to workers. If a worker's heart rate is abnormally high, the system will instruct them to stop work and take a break to protect their health.

[0361] Prompt Sentence Examples

[0362] Here are some example prompts to input to the generative AI model:

[0363] "Sensor data, camera footage, and audio data from within the factory are collected in real time, and workers' physiological data is analyzed to detect abnormalities. All data is analyzed using AI models, risk assessment is performed, and appropriate instructions are provided to workers."

[0364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0365] Step 1:

[0366] The server obtains flight data such as altitude, speed, geographic coordinates, and engine status from various sensors on the aircraft in real time. In addition, it also collects sensor data such as temperature, humidity, and machine operation status inside the factory. The input is sensor data, and the output is collected data in data frame format. The data is managed using the Pandas library.

[0367] Step 2:

[0368] The server acquires surveillance camera footage and factory audio data, and uses video and audio recognition technologies to detect abnormalities. The video data is analyzed using the OpenCV library, and the audio data is analyzed using the Librosa library. The input is the camera footage and audio data, and the output is a flag indicating whether or not an abnormality is present.

[0369] Step 3:

[0370] The server collects physiological data such as the worker's heart rate in real time and inputs it into the emotion engine. The emotion engine analyzes the biosignals and evaluates the worker's stress level. The input is physiological data and the output is stress level. Specifically, the emotional state is estimated from the heart rate and skin potential response.

[0371] Step 4:

[0372] The server inputs all collected flight data, video and audio data, sensor data, and physiological data into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors inside and outside the factory. The input is all collected data, and the output is the risk assessment results. The machine learning model is trained using Scikit-learn.

[0373] Step 5:

[0374] The server generates and notifies appropriate instructions to aircraft pilots and factory workers based on the analysis results. Instructions are provided in text and voice format. The input is the analysis result of the AI ​​model, and the output is the instruction content. Instructions are generated taking into account the results of the emotion engine.

[0375] Step 6:

[0376] The server detects abnormalities and emergencies in real time from the analysis data and activates the autopilot system or the factory's emergency response system as necessary. The input is real-time analysis data, and the output is the emergency state and the corresponding action. Specifically, when an abnormality is detected, the response system is immediately activated.

[0377] Step 7:

[0378] The server uses the collected data to continuously update the machine learning model, improving overall system performance. This allows new abnormal patterns to be quickly incorporated. The input is past collected data and new data, and the output is an updated AI model. New algorithms and parameter adjustments are also applied to the data as it is learned.

[0379] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0380] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0381] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0382] [Second embodiment]

[0383] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0384] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0385] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0386] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0387] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0388] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0389] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0390] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0391] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0392] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0393] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0394] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0395] ---

[0396] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0397] System Configuration

[0398] 1. Real-time data acquisition

[0399] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0400] 2. Analysis of video and audio data

[0401] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0402] 3. Data analysis using AI models

[0403] The server uses an AI model (machine learning algorithm) to analyze the flight data and analyzed video and audio data, thereby extracting risk factors related to flight safety and supporting appropriate decisions based on the analysis results.

[0404] 4. Providing instructions to pilots

[0405] The server then uses the analysis to provide instructions to the pilot in text and audio format, such as "maintain altitude" and "follow the designated route."

[0406] 5. Emergency Detection

[0407] The server uses the analyzed data to detect abnormalities and emergencies in real time, including engine malfunctions, sudden weather changes, and close encounters with other aircraft.

[0408] 6. Activating the autopilot system

[0409] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0410] 7. Continuous learning

[0411] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0412] ---

[0413] Processing flow and specific examples

[0414] Real-time data acquisition

[0415] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0416] Video and audio data analysis

[0417] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0418] Data analysis using AI models

[0419] The server inputs all acquired data into an AI model and extracts safety-related risk factors. For example, the analysis results show that sudden weather changes are predicted.

[0420] Providing instructions to pilots

[0421] Based on the analysis results, the server provides instructions to the pilot via text and voice notification, such as "Please change to a new route" to respond to weather changes.

[0422] Emergency Detection

[0423] The server analyzes flight data and detects emergencies such as engine malfunctions or sudden weather changes. For example, if an abnormally high engine temperature is detected, a notification will be sent.

[0424] Autopilot system activation

[0425] If an emergency is detected, the server will activate the autopilot system and take appropriate action, such as specifying an emergency landing route to an appropriate airport in the event of an engine malfunction.

[0426] Continuous learning

[0427] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns.

[0428] ---

[0429] As described above, the system of the present invention can improve aircraft safety and reduce the burden on pilots. This system utilizes advanced data analysis technology and machine learning to set a new standard in the aviation industry.

[0430] The processing flow will be explained below.

[0431] Step 1:

[0432] The server receives real-time flight data from various sensors on the plane. This data includes altitude, speed, geographic coordinates, and engine status. For example, if the plane is at an altitude of 10,000 feet and traveling at 550 knots, that data is sent to the server.

[0433] Step 2:

[0434] The server acquires video data from the airport. It collects real-time camera footage from within the airport and uses image recognition technology to analyze the runway conditions and the presence of obstacles in the surrounding area. For example, it can detect foreign objects on the runway from the camera footage.

[0435] Step 3:

[0436] The server acquires the voice data of the communication between the controller and the pilot and analyzes it using voice recognition technology. This detects erroneous instructions and mishearing by the pilot. For example, if the controller instructs "Turn right" but the pilot mistakenly interprets it as "Turn left," the system will detect the misunderstanding.

[0437] Step 4:

[0438] The server inputs the flight data, video data, and audio data it acquires into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors related to flight safety. For example, if a sudden change in weather is predicted, that information is extracted as a risk factor.

[0439] Step 5:

[0440] Based on the analysis results, the server generates appropriate instructions for the pilot. The generated instructions are displayed in text format on the cockpit display and also communicated to the pilot in voice format. For example, the instruction "Maintain altitude" is displayed and the voice also communicates "Maintain altitude."

[0441] Step 6:

[0442] The server detects emergencies from the analysis data, such as engine abnormalities, sudden weather changes, or close proximity to other aircraft. For example, if the engine temperature is abnormally high, it will be detected as an emergency.

[0443] Step 7:

[0444] When an emergency is detected, the server activates the autopilot system, which then executes the appropriate emergency response procedures. For example, in the event of an engine failure, the autopilot system calculates an emergency landing route to the nearest airport and autonomously directs the plane along that route.

[0445] Step 8:

[0446] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, new flight data can be used to refine the fuel efficiency optimization algorithm for the next flight.

[0447] These are the processing steps of the system of the present invention, including specific examples, which improves aircraft safety and reduces pilot workload.

[0448] Example 1

[0449] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0450] Improving safety and reducing pilot workload are key issues in modern aviation. In particular, failure to properly detect anomalies and respond to emergencies in real time poses a risk of serious accidents. Systems that efficiently analyze flight data and provide accurate flight instructions to pilots are also needed. Furthermore, continuous system performance improvement through machine learning is also necessary.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0452] In this invention, the server includes a means for acquiring flight data, a means for analyzing surveillance camera and wireless communication data, a means for generating aircraft operation instructions based on the analysis results, a means for detecting an emergency, a means for activating an autopilot system when the emergency is detected, and a means for continuously training a machine learning model using the flight data and analysis data. This enables abnormalities to be detected in real time and appropriate countermeasures to be implemented promptly. Furthermore, continuous learning improves the accuracy of the system, contributing to aircraft safety and reducing the burden on pilots.

[0453] "Flight data" is data that includes information such as aircraft altitude, speed, geographic coordinates, and engine status.

[0454] A "surveillance camera" is a device that captures images for the purpose of monitoring airports and aircraft.

[0455] "Radio communication data" refers to data of voice communications between air traffic controllers and pilots.

[0456] The "analysis results" are the analysis results obtained after analyzing the acquired data.

[0457] "Aircraft operation instructions" refers to specific instructions regarding the piloting and operation of an aircraft based on the analysis results.

[0458] An "emergency situation" is an abnormal condition that poses a significant risk to the safe operation of an aircraft.

[0459] An "autopilot system" is a system for automatically operating an aircraft.

[0460] A "machine learning model" refers to an algorithm or system that automatically learns from data and makes predictions and classifications.

[0461] "Continuous learning" is the process of constantly updating a model's learning with new data to improve its accuracy and performance.

[0462] MODE FOR CARRYING OUT THE INVENTION

[0463] The following describes in detail the mode for carrying out the present invention. The purpose of this system is to improve aircraft safety and reduce the burden on pilots. The hardware and software used, as well as the methods of data processing and calculation, are explained below.

[0464] System Configuration

[0465] Real-time data acquisition

[0466] The server obtains flight data in real time from various sensors on the aircraft. This flight data includes altitude, speed, geographic coordinates, and engine status. The server collects data using the aircraft's altimeter, speedometer, GPS, and engine monitoring system, and stores it in a database (e.g., MySQL) in real time.

[0467] As a specific example, the server obtains information in seconds about an airplane in flight, such as its altitude, speed, latitude, and longitude, which are 3,000 meters, 900 km / h, 35 degrees north, and 139 degrees east, and stores this information in a database.

[0468] Video and audio data analysis

[0469] The server collects and analyzes airport surveillance camera footage and radio communication data between controllers and pilots. Specifically, the server acquires surveillance camera footage and uses image recognition software (e.g., OpenCV) to extract important visual information. It also analyzes controller-pilot communications using voice recognition technology (e.g., Google Speech-to-Text API) to detect miscommunications and mishearing.

[0470] Specifically, the server detects moving objects on the runway from surveillance camera footage and uses voice recognition to confirm that the controller's instruction is to "climb 500 feet."

[0471] Data analysis using AI models

[0472] The server inputs the acquired data into a machine learning model for analysis. The server inputs flight data and video / audio analysis data acquired from the database into an AI model (e.g., TensorFlow), analyzes the data, and extracts potential risk factors. The analysis results are classified by risk type and sent to the server's internal evaluation system.

[0473] As a specific example, the server obtains predictions of sudden weather changes from an AI model and reflects this information in the risk assessment system.

[0474] Providing instructions to pilots

[0475] The server provides text and voice instructions to the pilot based on the analysis results. For example, the server checks the analysis results, generates text instructions, and displays them on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. A specific example of such an instruction would be "Please change to a new route" displayed on the screen and also given as a voice instruction.

[0476] Emergency Detection

[0477] The server detects emergencies based on real-time data analysis results. The server periodically monitors the analysis results and runs algorithms to detect abnormal patterns. If an emergency is detected, it immediately generates an alert and notifies relevant parties. Detailed information about the emergency is provided to the pilot.

[0478] As a specific example, the server detects an abnormal rise in engine temperature and notifies the pilot with an alert on the screen and via voice.

[0479] Autopilot system activation

[0480] When the server detects an emergency, it activates the autopilot system. When the server detects an emergency, it issues a command to activate the autopilot mode. It also sends a pre-defined emergency response route to the autopilot system and executes it. The autopilot system then initiates the designated emergency response.

[0481] As a specific example, if an engine malfunction is detected, the server will activate the autopilot system and route the aircraft to the nearest suitable airport for an emergency landing.

[0482] Continuous learning

[0483] The server continuously trains the AI ​​model using the acquired data, adds newly acquired flight data to the machine learning dataset, and periodically retrains the machine learning model to improve accuracy and performance, and deploys the updated model to the system.

[0484] For example, new flight data can be used to retrain AI models, improving the accuracy of anomaly detection.

[0485] Prompt Sentence Examples

[0486] "Please explain the specific processing flow of the system that acquires real-time data from the aircraft, analyzes video and audio to detect abnormalities, and issues instructions to the pilot. Please also explain the specific operation of each step."

[0487] As described above, the system of the present invention aims to improve aircraft safety and reduce pilot workload by utilizing advanced data analysis technology and machine learning, and this system can set a new standard in the aviation industry.

[0488] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0489] Program processing flow

[0490] Step 1:

[0491] The server acquires flight data in real time from various sensors on the aircraft. The input for this step is data from the altimeter, speedometer, GPS, and engine monitoring system installed on the aircraft. The server receives this data and stores it in a database (e.g., MySQL) in real time. Specifically, the server acquires data every second, such as an altitude of 3,000 meters, a speed of 900 km / h, latitude 35 degrees north, and longitude 139 degrees east, and stores it in the database.

[0492] Step 2:

[0493] The server collects and analyzes airport surveillance camera footage and radio communication data between the controller and pilot. The input for this step is live video data from the surveillance cameras and audio data from radio communication. The server uses OpenCV to analyze the video data and detect moving objects and anomalies. It also uses the Google Speech-to-Text API to convert audio data into text and detect erroneous instructions or mishearing. Specifically, the server detects foreign objects on the runway from surveillance camera footage and uses voice recognition to confirm the controller's instruction to "climb to an altitude of 500 feet."

[0494] Step 3:

[0495] The server inputs the acquired flight data and analyzed video and audio data into the AI ​​model for analysis. The inputs for this step are the flight data, video data, and audio data acquired in the previous step. The server uses TensorFlow to analyze this data and extract potential risk factors. The analysis results are classified by risk type and sent to the built-in evaluation system. Specifically, the server receives a result from the AI ​​model indicating that a sudden weather change is predicted, and reflects this information in the risk evaluation system.

[0496] Step 4:

[0497] Based on the analysis results, the server provides text and voice instructions to the pilot. The input for this step is the AI ​​analysis results. The server checks the analysis results, generates instructions, and displays them in text on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. Specifically, the instruction "Please change to a new route" is displayed on the screen and is also given voice instructions.

[0498] Step 5:

[0499] The server detects emergencies based on the results of real-time data analysis. The input to this step is the analysis results. The server periodically monitors the analysis results and runs an algorithm to detect abnormal patterns. If an emergency is detected, an alert is immediately generated and notified to the pilot. Specifically, the server detects an abnormal rise in engine temperature and notifies the pilot of the alert on the screen and by voice.

[0500] Step 6:

[0501] If the server detects an emergency, it activates the autopilot system. The input to this step is the emergency detection result. The server issues a command to activate the autopilot mode and sends a pre-set emergency response route to the autopilot system to execute. Specifically, if an engine abnormality is detected, the autopilot system is activated and an emergency landing route to the nearest appropriate airport is specified.

[0502] Step 7:

[0503] The server continuously trains the AI ​​model using the acquired data. The input for this step is newly acquired flight data. The server adds this to the machine learning dataset and periodically retrains the machine learning model to improve accuracy and performance. The updated model is then deployed to the system. Specifically, the AI ​​model is retrained using new flight data, improving the accuracy of anomaly detection.

[0504] Through the above processing steps, this system can improve aircraft safety and reduce the burden on pilots.

[0505] (Application example 1)

[0506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Operations at conventional logistics centers depended heavily on human labor, posing safety and efficiency challenges. For example, when equipment failure, abnormal temperature rises, or operational errors occurred, it was difficult to immediately detect and respond to these risks. Furthermore, due to insufficient real-time data analysis and automation, there were frequent delays in response in situations where appropriate decisions needed to be made immediately. Therefore, there was an urgent need to provide a system that would improve the safety and efficiency of logistics centers.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0509] In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data, means for acquiring various data within the logistics center in real time, means for providing instructions to workers and automated transport robots based on the analysis results, and means for automatically executing an emergency response in the event of an abnormality. This makes it possible to analyze real-time data from sensors and surveillance cameras in the logistics center and automate the provision of instructions to workers and automated transport robots and emergency responses.

[0510] "Flight data" refers to various data relating to the operation of an aircraft, including the aircraft's altitude, speed, geographic coordinates, engine status, etc.

[0511] "Video Data" means images or video information captured by surveillance cameras or other visual sensors.

[0512] "Voice data" means data containing communications between air traffic controllers and pilots and other voice information.

[0513] "Analysis results" refer to the results of analyzing the acquired data using AI models or machine learning algorithms.

[0514] "Aircraft operation instructions" refer to specific instructions regarding aircraft operation based on the analysis results, such as instructions to maintain altitude or change route.

[0515] An "emergency" is an abnormal condition, such as an engine malfunction or a sudden change in weather, that poses a significant risk to aircraft operation.

[0516] An "autopilot system" is an automated flight system that assists or substitutes for pilot operations under certain conditions, such as emergencies.

[0517] A "machine learning model" is an algorithm or program that continuously learns from collected data and improves the accuracy of analysis.

[0518] A "logistics center" is a facility that receives, stores, and ships goods.

[0519] "Real-time data" refers to data that is obtained instantly from sensors, surveillance cameras, etc. and processed without delay.

[0520] "Workers" refers to workers who work within a logistics center.

[0521] An "automatic transport robot" is a robot that performs automated transport tasks within a logistics center.

[0522] "Providing instructions" refers to the act of communicating the work content and response methods to workers and automatic transport robots based on the analysis results.

[0523] An "abnormal condition" is a deviation from normal operating conditions that may affect safety or efficiency.

[0524] "Emergency response" refers to immediate response measures taken in the event of an abnormality or emergency.

[0525] The following describes in detail a system for implementing an application example of the present invention.

[0526] System Configuration

[0527] This system is designed to support efficient and safe operations in logistics centers and primarily consists of a server, sensors, surveillance cameras, terminals (smartphones or tablets) used by workers, and an automated transport robot.

[0528] Hardware and Software Use

[0529] Hardware:

[0530] Various sensors (temperature, humidity, location information, etc.)

[0531] surveillance cameras

[0532] Smartphone or tablet

[0533] Automatic transport robot

[0534] software:

[0535] Python for Data Analysis (Flask Framework)

[0536] Machine learning models (scikit-learn and TensorFlow)

[0537] WebSocket (JavaScript) for real-time data acquisition

[0538] Data processing and calculation

[0539] The server receives real-time data from various sensors in the logistics center via WebSocket. The received data is analyzed using a Flask-based API. A pre-trained machine learning model (scikit-learn or TensorFlow) is used for the analysis, and this model detects risk factors such as temperature rises or abnormal machine operation.

[0540] Operations and Examples

[0541] Operation steps:

[0542] 1. Real-time data acquisition: Collect data such as temperature, humidity, and location information from sensors within the logistics center.

[0543] 2. Sending data: The collected data is sent to the server via WebSocket.

[0544] 3. Data analysis: The server's machine learning model analyzes the data and extracts risk factors.

[0545] 4. Providing instructions: Based on the analysis results, appropriate instructions are provided in voice and text format to the worker's device and the automatic transport robot.

[0546] 5. Response to abnormalities: If an abnormality is detected, an emergency response will be immediately implemented and the system will take autopilot.

[0547] Examples:

[0548] For example, if a temperature sensor detects an abnormal temperature rise in an area of ​​a logistics center, this data is sent to a server in real time. The server's machine learning model uses this information to determine that there is a high risk of fire. The system immediately provides a voice instruction to workers saying, "Abnormal temperature rise detected in Area B. Please check immediately," and instructs an automated transport robot to move from the area to a safe location.

[0549] Example prompt sentence:

[0550] Design an AI system to collect and analyze real-time data from various sensors in a logistics center. Based on the analysis results, provide instructions to workers and automated transport robots, and automatically respond in the event of an abnormality.

[0551] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0552] Step 1:

[0553] The server collects data in real time from various sensors in the logistics center. As input, it receives data such as temperature, humidity, and location information from each sensor, and as output, it stores this data in its internal storage.

[0554] Specifically, the system obtains the value measured by the temperature sensor (e.g., 25 degrees) and the coordinates obtained by the position sensor (e.g., 35 degrees north latitude, 139 degrees east longitude).

[0555] Step 2:

[0556] The server collects video data from the surveillance cameras and audio data from the microphones. As input, it receives video frames from the cameras and audio data from the microphones, and as output, it passes these data to the video analysis and audio analysis modules.

[0557] Specifically, it captures camera video frames (e.g., JPEG images with a resolution of 1920x1080) and records audio data (e.g., conversations between controllers and workers).

[0558] Step 3:

[0559] The server inputs the collected data into a machine learning model for analysis. Data from sensors and surveillance cameras is received as input, and risk factors are extracted as analysis results as output. Specific data processing, for example, involves analyzing temperature data over time to detect abnormal temperature increases.

[0560] Specific actions include identifying abnormal patterns (e.g., a temperature spike lasting 10 minutes) and reporting them as a risk.

[0561] Step 4:

[0562] The server provides instructions to workers and automated transport robots based on the analysis results. It receives the risk factors of the analysis results as input, generates instructions in text and voice format as output, and sends them to the workers' terminals and the robots.

[0563] Specifically, the system sends a voice message to the worker's smartphone saying, "An abnormal temperature rise has been detected in Area B. Please check immediately," and instructs the robot to move to a safe location.

[0564] Step 5:

[0565] The server automatically executes emergency response when an anomaly is detected. It receives anomaly detection alerts as input and executes specific response measures as output.

[0566] Specifically, if it is determined that there is a high risk of fire, the system will immediately contact the fire department and move the automated transport robot to a safe location.

[0567] Step 6:

[0568] The server continuously trains the machine learning model using data collected in real time, taking newly collected data as input and improving the model's performance as output.

[0569] Specifically, the system uses previously collected data on normal and abnormal patterns as training data and updates the model to improve analysis accuracy.

[0570] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0571] ---

[0572] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0573] System Configuration

[0574] 1. Real-time data acquisition

[0575] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0576] 2. Analysis of video and audio data

[0577] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0578] 3. User Emotion Recognition

[0579] The server provides the pilot's voice and biological information to the emotion engine, which analyzes the user's (pilot's) emotions. The emotion engine recognizes the pilot's emotional state from voice tone and physiological data (heart rate, skin potential response, etc.).

[0580] 4. Data analysis using AI models

[0581] The server inputs the flight data, video data, audio data, and emotion data into the AI ​​model for analysis. The AI ​​model integrates this data and extracts risk factors related to flight safety.

[0582] 5. Providing instructions to pilots

[0583] The server then uses the analysis results to provide instructions to the pilot in text and voice format. Using the results of the emotion engine, the server generates appropriate instructions based on the pilot's emotional state. For example, if the pilot is under high stress, the server provides concise and clear instructions, adding reassuring words.

[0584] 6. Emergency Detection

[0585] The server uses the analysis data to detect abnormal situations and emergencies in real time. It also takes into account data from the emotion engine to more accurately detect emergencies. For example, if a pilot's heart rate suddenly rises or their emotional state suddenly changes, it will notify them of that condition as a warning.

[0586] 7. Activating the autopilot system

[0587] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0588] 8. Continuous learning

[0589] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0590] ---

[0591] Processing flow and specific examples

[0592] Real-time data acquisition

[0593] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0594] Video and audio data analysis

[0595] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0596] User Emotion Recognition

[0597] The server analyzes the pilot's emotional state using an emotion engine. Based on collected physiological data such as voice tone and heart rate, it determines whether the pilot is under stress. For example, if the heart rate is abnormally high and the voice tone is tense, it determines that the pilot is under high stress.

[0598] Data analysis using AI models

[0599] The server inputs all acquired data into an AI model to extract safety-related risk factors. For example, the analysis results show that a sudden change in weather is predicted.

[0600] Providing instructions to pilots

[0601] Based on the analysis results, the server provides instructions to the pilot via text display and voice notification. The emotion engine's analysis results are reflected in the instructions generated based on the pilot's emotional state. For example, in a high-stress situation, additional instructions such as "Remain calm and maintain your current altitude" are given.

[0602] Emergency Detection

[0603] The server analyzes flight data to detect emergencies such as engine malfunctions or sudden weather changes. It also takes into account data from the emotion engine, and if an abnormal physiological state of the pilot is observed, it uses that information to issue a highly accurate warning. For example, if the engine temperature is abnormally high and the pilot's heart rate is also rising sharply, this will be detected as an emergency.

[0604] Autopilot system activation

[0605] If the server detects an emergency, it immediately activates the autopilot system and instructs the appropriate emergency response. For example, in the event of an engine malfunction, it will instruct the aircraft to make an emergency landing at an appropriate airport, and the autopilot system will fly that route autonomously.

[0606] Continuous learning

[0607] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns, for example, by improving the system's response algorithms based on newly observed emergency patterns.

[0608] The above are the processing steps of the system of the present invention, including specific examples. By combining it with an emotion engine, the safety of aircraft can be further improved and the mental burden on pilots can be effectively reduced.

[0609] The processing flow will be explained below.

[0610] Step 1:

[0611] The server receives real-time flight data from various sensors on the aircraft, including altitude, speed, geographic coordinates, and engine status. For example, the flight altitude is 30,000 feet, the speed is 500 knots, the geographic coordinates are 35 degrees north, 140 degrees east, and the engine status is normal.

[0612] Step 2:

[0613] The server receives camera footage from the airport. The video data is analyzed using image recognition technology to detect obstacles and abnormalities on the runway. For example, if a bird is detected on the runway through video analysis, it will identify it and issue a warning.

[0614] Step 3:

[0615] The server receives voice data from radio communications between the controller and the pilot and analyzes it using voice recognition technology. For example, it checks whether the controller's instructions, such as "Maintain current altitude," were properly conveyed to the pilot.

[0616] Step 4:

[0617] The server provides the pilot's voice, heart rate, and other physiological information to the emotion engine, which analyzes the user's (pilot's) emotional state. The emotion engine recognizes the pilot's stress and tension from the tone of their voice and heart rate. For example, if their heart rate is higher than normal and their voice tone is rising, it determines that the pilot is nervous.

[0618] Step 5:

[0619] The server inputs flight data, video data, audio data, and emotion data into the AI ​​model, and analyzes risk factors related to flight safety. For example, sudden weather changes and close proximity to other aircraft are extracted as risk factors.

[0620] Step 6:

[0621] The server generates appropriate instructions for the pilot based on the analysis results. It also reflects the results of the emotion engine and generates instructions that correspond to the pilot's emotional state. For example, it may say, "Please relax. You are instructed to maintain your current altitude."

[0622] Step 7:

[0623] The server detects emergencies from the analyzed data, including engine abnormalities, sudden weather changes, close encounters with other aircraft, etc. For example, if the engine temperature rises sharply and the pilot's heart rate also increases, this will be detected as an emergency.

[0624] Step 8:

[0625] If an emergency is detected, the server immediately activates the autopilot system and instructs the necessary emergency response. For example, in the event of an engine malfunction, the autopilot system can set up an emergency landing route to the nearest safe airport and fly that route autonomously.

[0626] Step 9:

[0627] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, it can learn new emergency patterns and improve prediction accuracy for the next flight.

[0628] The above are the detailed processing steps of the system of the present invention that combines the emotion engine. This is expected to further improve aircraft safety and effectively reduce the mental burden on pilots.

[0629] Example 2

[0630] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] Current aircraft systems face the problem of not being able to ensure flight safety or sufficiently reduce the pilot's mental burden by simply analyzing flight, video, and audio information. Failure to detect emergencies and provide appropriate instructions to pilots in real time increases the risk of serious accidents. Furthermore, there is no way to provide instructions that take into account the pilot's emotional state and stress level, which does not reduce the pilot's mental burden. A new system is needed to solve these issues and achieve safer and more efficient flight.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0633] In this invention, the server includes means for acquiring flight information, means for analyzing video and audio information, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight information, means for analyzing the emotional state of the pilot from physiological information and audio information, and means for generating appropriate aircraft operation instructions using the emotional state analysis results, thereby improving flight safety and reducing the mental burden on the pilot.

[0634] "Flight information" refers to real-time data such as an aircraft's altitude, speed, geographic coordinates, and engine status.

[0635] "Video and audio information" refers to digital video and audio data obtained from airport surveillance camera footage and radio communications with air traffic controllers.

[0636] "Analysis results" refers to the data obtained by processing flight information, video, and audio information, and specifically includes the detection of erroneous instructions and the extraction of risk factors.

[0637] "Aircraft operation instructions" refer to operational guidance and action instructions for the pilot that are generated based on the analysis results.

[0638] An "emergency" refers to a situation that threatens the safety of an aircraft, such as an engine malfunction, a sudden change in weather, or a physiological abnormality in the pilot.

[0639] An "autopilot system" is a system that operates an aircraft autonomously without pilot intervention in the event of an emergency.

[0640] "Machine learning model" refers to a statistical model that uses algorithms to accumulate knowledge and continuously learn, and includes models that make predictions and decisions based on flight information.

[0641] "Physiological information" refers to data that indicates the pilot's physical condition, such as their heart rate and skin potential response.

[0642] "Emotional state" refers to the pilot's psychological state as analyzed from their vocal tone and physiological information.

[0643] "Emotional state analysis results" refers to data on the pilot's emotional state analyzed by the emotion engine, including stress levels and tension.

[0644] "Appropriate aircraft operation instructions" refers to operational guidance and behavioral instructions that correspond to the pilot's mental condition and are generated based on the results of emotional state analysis.

[0645] MODE FOR CARRYING OUT THE INVENTION

[0646] An embodiment of the system of the present invention is described in detail below.

[0647] This system aims to improve aircraft safety by collecting and analyzing aircraft flight data, video and audio data, and physiological information. In particular, it analyzes the pilot's emotional state and generates appropriate instructions, thereby reducing the pilot's mental burden.

[0648] The server receives real-time flight data from the plane's various sensors. This includes the plane's altitude, speed, geographic coordinates, and engine status. For example, altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east. This data is retrieved from the aircraft's central system using a dedicated protocol.

[0649] The server collects video and audio data from airport surveillance cameras and wireless communications with air traffic controllers. Video data is acquired as streaming from the surveillance cameras, and audio data is received in digital audio format. Specifically, the system uses video analysis algorithms to detect foreign objects on the runway and voice recognition technology to analyze air traffic controllers' instructions in real time to check for erroneous instructions.

[0650] The server provides the pilot's voice and biometric information to the emotion engine, which analyzes their emotional state. The emotion engine analyzes their voice tone, heart rate, skin potential response, and other factors to recognize their emotional state. Data is collected from the audio microphone and biometric sensors and processed in real time. For example, if the pilot's heart rate is 30% higher than normal and their voice tone sounds tense, it is determined that the pilot is experiencing high levels of stress.

[0651] All data collected by the server (flight data, video data, audio data, emotion data) is input into the AI ​​model for analysis. The AI ​​model is composed of multiple neural networks and integrates and analyzes data. For example, sudden changes in weather are extracted as analysis results and reported as risk factors.

[0652] Based on the analysis results, the server provides instructions to the pilot in text and voice format. The instructions are generated based on the results of the emotion engine. For example, a pilot in a state of high stress may be given voice instructions such as "Remain calm and maintain your current altitude," which are also displayed on the screen in text format.

[0653] The server analyzes flight data in real time to detect abnormalities and emergencies. Emergencies include engine malfunctions, sudden weather changes, and abnormal pilot physiological states. For example, if the engine temperature suddenly rises and the pilot's heart rate also rises sharply, this will be detected as an emergency and an alert will be sent immediately.

[0654] When the server detects an emergency, it immediately activates the autopilot system, which executes emergency responses based on pre-programmed scenarios. For example, if an engine malfunction is detected, the autopilot system will execute an emergency landing route to the nearest airport.

[0655] The server continuously collects flight data and periodically updates the AI ​​model. This update process is performed offline, and new data sets are used to improve the model's performance. For example, new emergency scenarios can be learned and the system can adapt to them.

[0656] Prompt Sentence Examples

[0657] "The pilot's heart rate has been consistently 30% higher than normal. Please provide a specific example of what instructions should be given to the pilot if this condition is detected."

[0658] The above is an embodiment of the present invention, which can improve the safety of aircraft and effectively reduce the mental burden on pilots.

[0659] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0660] Step 1:

[0661] The server obtains flight data in real time from the aircraft's sensors. Specifically, data such as the aircraft's altitude, speed, geographic coordinates, and engine status are obtained from the aircraft's central system using a dedicated protocol. The input data is raw data from the sensors, which is collected in real time and stored in a database. As an output, various flight data are stored in the database in an organized manner.

[0662] Step 2:

[0663] The server acquires video data from surveillance cameras within the airport and collects audio data from radio communications with air traffic controllers. The video data is received in streaming format, and the audio data is received in digital audio format. The video data is run through an image recognition algorithm to detect foreign objects on the runway, and the audio data is run through voice recognition technology to check for erroneous instructions. The output is compiled into a report containing the foreign object detection results and audio analysis results.

[0664] Step 3:

[0665] The server collects the pilot's voice and physiological information and provides it to the emotion engine. Voice is captured through a microphone, and physiological data is acquired through a heart rate sensor and skin potential sensor. The emotion engine analyzes this data and recognizes the pilot's emotional state. The input is the voice and physiological data collected from the pilot, and the output is the pilot's emotional state (e.g., high stress, relaxed).

[0666] Step 4:

[0667] The flight data, video data, audio data, and emotional data acquired by the server are input into the AI ​​model. The AI ​​model includes a neural network that integrates and analyzes this data. The input data is initial data from various sensors, and the output is a report containing the integrated analysis results (e.g., sudden weather changes and risk factors). Specifically, the model performs functions such as detecting anomalies in flight data, detecting foreign objects in video data, and detecting erroneous instructions in audio data.

[0668] Step 5:

[0669] Based on the analysis results, the server provides instructions to the pilot in text and voice format. Instructions are generated based on the results of the emotion engine, and the pilot is given appropriate operational guidance. Analysis results and emotion data are used as input data, and the output is voice instructions (e.g., "Please remain calm and maintain your current altitude") and text instructions displayed on the pilot's display.

[0670] Step 6:

[0671] The server analyzes flight data and emotional data in real time to detect emergencies. If an engine abnormality, a sudden change in weather, or a pilot's physiological abnormality is detected, an emergency alert is issued immediately. The input is real-time flight data and emotional data, and the output is a warning message about the emergency and detailed information.

[0672] Step 7:

[0673] When the server detects an emergency, it activates the autopilot system. The autopilot system then takes appropriate action based on pre-programmed emergency response scenarios. For example, if an engine malfunction is detected, an emergency landing route to the nearest airport is automatically set and the flight is carried out along that route. The input is the emergency detection information, and the output is the activation of the autopilot system and a log of its actions.

[0674] Step 8:

[0675] The server continuously accumulates collected flight data and uses it to improve the performance of the AI ​​model. The newly acquired data is used to retrain the model, enabling more accurate analysis. The input is continuously collected flight data, and the output is an updated AI model and its performance evaluation results. For example, it is possible to learn new patterns of abnormal situations and incorporate appropriate countermeasures into the system.

[0676] (Application example 2)

[0677] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] In modern manufacturing, improving the efficiency and safety of factory operations are important challenges. In particular, there is a need to monitor the work environment and worker status in real time and respond quickly and appropriately when an abnormality occurs. However, existing systems lack the technology to integrate and analyze multiple data sources and automatically generate work instructions, leaving room for improvement. Therefore, there is a need for a system that can efficiently integrate and analyze real-time data and provide work instructions, including emergency responses.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data and environmental data, means for acquiring sensor data within the factory, means for analyzing environmental data and physiological data, means for generating work instructions based on the analysis results, means for autonomously responding to emergencies based on the results of the analysis means, and means for updating the generating AI model and optimizing the instruction provision. This enables efficient work management and improved safety within the factory.

[0680] "Flight data" is data including aircraft altitude, speed, geographic coordinates, and engine status.

[0681] "Video and audio data" refers to data that includes video and audio from a surveillance camera.

[0682] "Aircraft operation instructions" are instructions regarding flight operations sent from the server to the pilot.

[0683] "Emergency" means an abnormal or dangerous condition in aircraft operations or factory operations.

[0684] An "autopilot system" is a system that controls the automatic operation of an aircraft.

[0685] A "machine learning model" is an artificial intelligence model that is trained to perform analysis and predictions using collected data.

[0686] "Sensor data" is data obtained from various sensors within the factory (temperature, humidity, machine operating status, etc.).

[0687] "Environmental data" refers to data including environmental conditions such as temperature, humidity, noise, and vibration within a factory.

[0688] "Physiological data" refers to data that indicates the physiological state of a worker, such as heart rate and skin potential.

[0689] "Work instructions" are specific instructions for actions provided to workers based on the analysis results.

[0690] A "generative AI model" is an artificial intelligence model that generates new instructions and countermeasures by analyzing data.

[0691] "Autonomous emergency response" means that the system automatically takes action to deal with an emergency.

[0692] As an embodiment of the present invention, the system is described in detail below. This system collects and analyzes flight data, video and audio data, sensor data, environmental data, physiological data, etc. in real time, and issues appropriate instructions and emergency responses based on the analysis results.

[0693] System Configuration

[0694] 1. Real-time data acquisition

[0695] The server receives real-time flight data from various sensors on the plane, including altitude, speed, geographic coordinates, and engine status, as well as sensor data from the factory, such as temperature, humidity, and machine operation status.

[0696] 2. Analysis of video and audio data

[0697] The server acquires surveillance camera footage and environmental audio, and uses video recognition technology to extract important visual information and audio recognition technology to detect signs of abnormalities.

[0698] 3. Physiological Data Collection and Analysis

[0699] The server collects physiological data such as the worker's heart rate and provides it to the emotion engine to analyze the worker's stress level. This analysis determines the worker's workload and takes the necessary measures.

[0700] 4. Data analysis using AI models

[0701] The server inputs the acquired flight data, video and audio data, sensor data, and physiological data into the AI ​​model, which then integrates and analyzes this data to identify risk factors both inside and outside the factory, contributing to improved safety and efficiency.

[0702] 5. Instruction Generation and Delivery

[0703] Based on the analysis, the server generates appropriate instructions for an aircraft pilot or a factory worker, delivered in text or audio format and including specific advice based on the worker's emotional state.

[0704] 6. Emergency Detection and Autonomous Response

[0705] The server uses the analysis data to detect abnormalities and emergencies in real time, and activates the autopilot system or the factory's emergency response system as necessary, allowing for a quick and safe response.

[0706] 7. Continuous learning

[0707] The server continuously updates the machine learning model using the collected data to improve the performance of the entire system, allowing it to quickly incorporate new abnormal patterns.

[0708] Hardware and software used

[0709] The hardware used is as follows:

[0710] Sensors: Various sensors that collect environmental data within the factory in real time

[0711] Camera: A surveillance camera for capturing video data

[0712] Physiological data monitor: A monitor that acquires information such as the worker's heart rate

[0713] The software used is as follows:

[0714] Pandas: Real-time data acquisition in dataframe format

[0715] OpenCV: Camera image analysis

[0716] Librosa: Analysis of audio data

[0717] Scikit-learn: Building and analyzing machine learning models

[0718] Specific examples

[0719] For example, if a factory machine detects abnormal vibrations, the system will immediately analyze the vibration data and provide appropriate instructions to workers. If a worker's heart rate is abnormally high, the system will instruct them to stop work and take a break to protect their health.

[0720] Prompt Sentence Examples

[0721] Here are some example prompts to input to the generative AI model:

[0722] "Sensor data, camera footage, and audio data from within the factory are collected in real time, and workers' physiological data is analyzed to detect abnormalities. All data is analyzed using AI models, risk assessment is performed, and appropriate instructions are provided to workers."

[0723] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0724] Step 1:

[0725] The server obtains flight data such as altitude, speed, geographic coordinates, and engine status from various sensors on the aircraft in real time. In addition, it also collects sensor data such as temperature, humidity, and machine operation status inside the factory. The input is sensor data, and the output is collected data in data frame format. The data is managed using the Pandas library.

[0726] Step 2:

[0727] The server acquires surveillance camera footage and factory audio data, and uses video and audio recognition technologies to detect abnormalities. The video data is analyzed using the OpenCV library, and the audio data is analyzed using the Librosa library. The input is the camera footage and audio data, and the output is a flag indicating whether or not an abnormality is present.

[0728] Step 3:

[0729] The server collects physiological data such as the worker's heart rate in real time and inputs it into the emotion engine. The emotion engine analyzes the biosignals and evaluates the worker's stress level. The input is physiological data and the output is stress level. Specifically, the emotional state is estimated from the heart rate and skin potential response.

[0730] Step 4:

[0731] The server inputs all collected flight data, video and audio data, sensor data, and physiological data into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors inside and outside the factory. The input is all collected data, and the output is the risk assessment results. The machine learning model is trained using Scikit-learn.

[0732] Step 5:

[0733] The server generates and notifies appropriate instructions to aircraft pilots and factory workers based on the analysis results. Instructions are provided in text and voice format. The input is the analysis result of the AI ​​model, and the output is the instruction content. Instructions are generated taking into account the results of the emotion engine.

[0734] Step 6:

[0735] The server detects abnormalities and emergencies in real time from the analysis data and activates the autopilot system or the factory's emergency response system as necessary. The input is real-time analysis data, and the output is the emergency state and the corresponding action. Specifically, when an abnormality is detected, the response system is immediately activated.

[0736] Step 7:

[0737] The server uses the collected data to continuously update the machine learning model, improving overall system performance. This allows new abnormal patterns to be quickly incorporated. The input is past collected data and new data, and the output is an updated AI model. New algorithms and parameter adjustments are also applied to the data as it is learned.

[0738] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0740] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0741] [Third embodiment]

[0742] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0743] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0744] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0745] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0746] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0747] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0748] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0749] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0750] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0751] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0752] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0753] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0754] ---

[0755] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0756] System Configuration

[0757] 1. Real-time data acquisition

[0758] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0759] 2. Analysis of video and audio data

[0760] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0761] 3. Data analysis using AI models

[0762] The server uses an AI model (machine learning algorithm) to analyze the flight data and analyzed video and audio data, thereby extracting risk factors related to flight safety and supporting appropriate decisions based on the analysis results.

[0763] 4. Providing instructions to pilots

[0764] The server then uses the analysis to provide instructions to the pilot in text and audio format, such as "maintain altitude" and "follow the designated route."

[0765] 5. Emergency Detection

[0766] The server uses the analyzed data to detect abnormalities and emergencies in real time, including engine malfunctions, sudden weather changes, and close encounters with other aircraft.

[0767] 6. Activating the autopilot system

[0768] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0769] 7. Continuous learning

[0770] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0771] ---

[0772] Processing flow and specific examples

[0773] Real-time data acquisition

[0774] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0775] Video and audio data analysis

[0776] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0777] Data analysis using AI models

[0778] The server inputs all acquired data into an AI model and extracts safety-related risk factors. For example, the analysis results show that sudden weather changes are predicted.

[0779] Providing instructions to pilots

[0780] Based on the analysis results, the server provides instructions to the pilot via text and voice notification, such as "Please change to a new route" to respond to weather changes.

[0781] Emergency Detection

[0782] The server analyzes flight data and detects emergencies such as engine malfunctions or sudden weather changes. For example, if an abnormally high engine temperature is detected, a notification will be sent.

[0783] Autopilot system activation

[0784] If an emergency is detected, the server will activate the autopilot system and take appropriate action, such as specifying an emergency landing route to an appropriate airport in the event of an engine malfunction.

[0785] Continuous learning

[0786] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns.

[0787] ---

[0788] As described above, the system of the present invention can improve aircraft safety and reduce the burden on pilots. This system utilizes advanced data analysis technology and machine learning to set a new standard in the aviation industry.

[0789] The processing flow will be explained below.

[0790] Step 1:

[0791] The server receives real-time flight data from various sensors on the plane. This data includes altitude, speed, geographic coordinates, and engine status. For example, if the plane is at an altitude of 10,000 feet and traveling at 550 knots, that data is sent to the server.

[0792] Step 2:

[0793] The server acquires video data from the airport. It collects real-time camera footage from within the airport and uses image recognition technology to analyze the runway conditions and the presence of obstacles in the surrounding area. For example, it can detect foreign objects on the runway from the camera footage.

[0794] Step 3:

[0795] The server acquires the voice data of the communication between the controller and the pilot and analyzes it using voice recognition technology. This detects erroneous instructions and mishearing by the pilot. For example, if the controller instructs "Turn right" but the pilot mistakenly interprets it as "Turn left," the system will detect the misunderstanding.

[0796] Step 4:

[0797] The server inputs the flight data, video data, and audio data it acquires into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors related to flight safety. For example, if a sudden change in weather is predicted, that information is extracted as a risk factor.

[0798] Step 5:

[0799] Based on the analysis results, the server generates appropriate instructions for the pilot. The generated instructions are displayed in text format on the cockpit display and also communicated to the pilot in voice format. For example, the instruction "Maintain altitude" is displayed and the voice also communicates "Maintain altitude."

[0800] Step 6:

[0801] The server detects emergencies from the analysis data, such as engine abnormalities, sudden weather changes, or close proximity to other aircraft. For example, if the engine temperature is abnormally high, it will be detected as an emergency.

[0802] Step 7:

[0803] When an emergency is detected, the server activates the autopilot system, which then executes the appropriate emergency response procedures. For example, in the event of an engine failure, the autopilot system calculates an emergency landing route to the nearest airport and autonomously directs the plane along that route.

[0804] Step 8:

[0805] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, new flight data can be used to refine the fuel efficiency optimization algorithm for the next flight.

[0806] These are the processing steps of the system of the present invention, including specific examples, which improves aircraft safety and reduces pilot workload.

[0807] Example 1

[0808] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0809] Improving safety and reducing pilot workload are key issues in modern aviation. In particular, failure to properly detect anomalies and respond to emergencies in real time poses a risk of serious accidents. Systems that efficiently analyze flight data and provide accurate flight instructions to pilots are also needed. Furthermore, continuous system performance improvement through machine learning is also necessary.

[0810] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0811] In this invention, the server includes a means for acquiring flight data, a means for analyzing surveillance camera and wireless communication data, a means for generating aircraft operation instructions based on the analysis results, a means for detecting an emergency, a means for activating an autopilot system when the emergency is detected, and a means for continuously training a machine learning model using the flight data and analysis data. This enables abnormalities to be detected in real time and appropriate countermeasures to be implemented promptly. Furthermore, continuous learning improves the accuracy of the system, contributing to aircraft safety and reducing the burden on pilots.

[0812] "Flight data" is data that includes information such as aircraft altitude, speed, geographic coordinates, and engine status.

[0813] A "surveillance camera" is a device that captures images for the purpose of monitoring airports and aircraft.

[0814] "Radio communication data" refers to data of voice communications between air traffic controllers and pilots.

[0815] The "analysis results" are the analysis results obtained after analyzing the acquired data.

[0816] "Aircraft operation instructions" refers to specific instructions regarding the piloting and operation of an aircraft based on the analysis results.

[0817] An "emergency situation" is an abnormal condition that poses a significant risk to the safe operation of an aircraft.

[0818] An "autopilot system" is a system for automatically operating an aircraft.

[0819] A "machine learning model" refers to an algorithm or system that automatically learns from data and makes predictions and classifications.

[0820] "Continuous learning" is the process of constantly updating a model's learning with new data to improve its accuracy and performance.

[0821] MODE FOR CARRYING OUT THE INVENTION

[0822] The following describes in detail the mode for carrying out the present invention. The purpose of this system is to improve aircraft safety and reduce the burden on pilots. The hardware and software used, as well as the methods of data processing and calculation, are explained below.

[0823] System Configuration

[0824] Real-time data acquisition

[0825] The server obtains flight data in real time from various sensors on the aircraft. This flight data includes altitude, speed, geographic coordinates, and engine status. The server collects data using the aircraft's altimeter, speedometer, GPS, and engine monitoring system, and stores it in a database (e.g., MySQL) in real time.

[0826] As a specific example, the server obtains information in seconds about an airplane in flight, such as its altitude, speed, latitude, and longitude, which are 3,000 meters, 900 km / h, 35 degrees north, and 139 degrees east, and stores this information in a database.

[0827] Video and audio data analysis

[0828] The server collects and analyzes airport surveillance camera footage and radio communication data between controllers and pilots. Specifically, the server acquires surveillance camera footage and uses image recognition software (e.g., OpenCV) to extract important visual information. It also analyzes controller-pilot communications using voice recognition technology (e.g., Google Speech-to-Text API) to detect miscommunications and mishearing.

[0829] Specifically, the server detects moving objects on the runway from surveillance camera footage and uses voice recognition to confirm that the controller's instruction is to "climb 500 feet."

[0830] Data analysis using AI models

[0831] The server inputs the acquired data into a machine learning model for analysis. The server inputs flight data and video / audio analysis data acquired from the database into an AI model (e.g., TensorFlow), analyzes the data, and extracts potential risk factors. The analysis results are classified by risk type and sent to the server's internal evaluation system.

[0832] As a specific example, the server obtains predictions of sudden weather changes from an AI model and reflects this information in the risk assessment system.

[0833] Providing instructions to pilots

[0834] The server provides text and voice instructions to the pilot based on the analysis results. For example, the server checks the analysis results, generates text instructions, and displays them on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. A specific example of such an instruction would be "Please change to a new route" displayed on the screen and also given as a voice instruction.

[0835] Emergency Detection

[0836] The server detects emergencies based on real-time data analysis results. The server periodically monitors the analysis results and runs algorithms to detect abnormal patterns. If an emergency is detected, it immediately generates an alert and notifies relevant parties. Detailed information about the emergency is provided to the pilot.

[0837] As a specific example, the server detects an abnormal rise in engine temperature and notifies the pilot with an alert on the screen and via voice.

[0838] Autopilot system activation

[0839] When the server detects an emergency, it activates the autopilot system. When the server detects an emergency, it issues a command to activate the autopilot mode. It also sends a pre-defined emergency response route to the autopilot system and executes it. The autopilot system then initiates the designated emergency response.

[0840] As a specific example, if an engine malfunction is detected, the server will activate the autopilot system and route the aircraft to the nearest suitable airport for an emergency landing.

[0841] Continuous learning

[0842] The server continuously trains the AI ​​model using the acquired data, adds newly acquired flight data to the machine learning dataset, and periodically retrains the machine learning model to improve accuracy and performance, and deploys the updated model to the system.

[0843] For example, new flight data can be used to retrain AI models, improving the accuracy of anomaly detection.

[0844] Prompt Sentence Examples

[0845] "Please explain the specific processing flow of the system that acquires real-time data from the aircraft, analyzes video and audio to detect abnormalities, and issues instructions to the pilot. Please also explain the specific operation of each step."

[0846] As described above, the system of the present invention aims to improve aircraft safety and reduce pilot workload by utilizing advanced data analysis technology and machine learning, and this system can set a new standard in the aviation industry.

[0847] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0848] Program processing flow

[0849] Step 1:

[0850] The server acquires flight data in real time from various sensors on the aircraft. The input for this step is data from the altimeter, speedometer, GPS, and engine monitoring system installed on the aircraft. The server receives this data and stores it in a database (e.g., MySQL) in real time. Specifically, the server acquires data every second, such as an altitude of 3,000 meters, a speed of 900 km / h, latitude 35 degrees north, and longitude 139 degrees east, and stores it in the database.

[0851] Step 2:

[0852] The server collects and analyzes airport surveillance camera footage and radio communication data between the controller and pilot. The input for this step is live video data from the surveillance cameras and audio data from radio communication. The server uses OpenCV to analyze the video data and detect moving objects and anomalies. It also uses the Google Speech-to-Text API to convert audio data into text and detect erroneous instructions or mishearing. Specifically, the server detects foreign objects on the runway from surveillance camera footage and uses voice recognition to confirm the controller's instruction to "climb to an altitude of 500 feet."

[0853] Step 3:

[0854] The server inputs the acquired flight data and analyzed video and audio data into the AI ​​model for analysis. The inputs for this step are the flight data, video data, and audio data acquired in the previous step. The server uses TensorFlow to analyze this data and extract potential risk factors. The analysis results are classified by risk type and sent to the built-in evaluation system. Specifically, the server receives a result from the AI ​​model indicating that a sudden weather change is predicted, and reflects this information in the risk evaluation system.

[0855] Step 4:

[0856] Based on the analysis results, the server provides text and voice instructions to the pilot. The input for this step is the AI ​​analysis results. The server checks the analysis results, generates instructions, and displays them in text on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. Specifically, the instruction "Please change to a new route" is displayed on the screen and is also given voice instructions.

[0857] Step 5:

[0858] The server detects emergencies based on the results of real-time data analysis. The input to this step is the analysis results. The server periodically monitors the analysis results and runs an algorithm to detect abnormal patterns. If an emergency is detected, an alert is immediately generated and notified to the pilot. Specifically, the server detects an abnormal rise in engine temperature and notifies the pilot of the alert on the screen and by voice.

[0859] Step 6:

[0860] If the server detects an emergency, it activates the autopilot system. The input to this step is the emergency detection result. The server issues a command to activate the autopilot mode and sends a pre-set emergency response route to the autopilot system to execute. Specifically, if an engine abnormality is detected, the autopilot system is activated and an emergency landing route to the nearest appropriate airport is specified.

[0861] Step 7:

[0862] The server continuously trains the AI ​​model using the acquired data. The input for this step is newly acquired flight data. The server adds this to the machine learning dataset and periodically retrains the machine learning model to improve accuracy and performance. The updated model is then deployed to the system. Specifically, the AI ​​model is retrained using new flight data, improving the accuracy of anomaly detection.

[0863] Through the above processing steps, this system can improve aircraft safety and reduce the burden on pilots.

[0864] (Application example 1)

[0865] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0866] Operations at conventional logistics centers depended heavily on human labor, posing safety and efficiency challenges. For example, when equipment failure, abnormal temperature rises, or operational errors occurred, it was difficult to immediately detect and respond to these risks. Furthermore, due to insufficient real-time data analysis and automation, there were frequent delays in response in situations where appropriate decisions needed to be made immediately. Therefore, there was an urgent need to provide a system that would improve the safety and efficiency of logistics centers.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0868] In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data, means for acquiring various data within the logistics center in real time, means for providing instructions to workers and automated transport robots based on the analysis results, and means for automatically executing an emergency response in the event of an abnormality. This makes it possible to analyze real-time data from sensors and surveillance cameras in the logistics center and automate the provision of instructions to workers and automated transport robots and emergency responses.

[0869] "Flight data" refers to various data relating to the operation of an aircraft, including the aircraft's altitude, speed, geographic coordinates, engine status, etc.

[0870] "Video Data" means images or video information captured by surveillance cameras or other visual sensors.

[0871] "Voice data" means data containing communications between air traffic controllers and pilots and other voice information.

[0872] "Analysis results" refer to the results of analyzing the acquired data using AI models or machine learning algorithms.

[0873] "Aircraft operation instructions" refer to specific instructions regarding aircraft operation based on the analysis results, such as instructions to maintain altitude or change route.

[0874] An "emergency" is an abnormal condition, such as an engine malfunction or a sudden change in weather, that poses a significant risk to aircraft operation.

[0875] An "autopilot system" is an automated flight system that assists or substitutes for pilot operations under certain conditions, such as emergencies.

[0876] A "machine learning model" is an algorithm or program that continuously learns from collected data and improves the accuracy of analysis.

[0877] A "logistics center" is a facility that receives, stores, and ships goods.

[0878] "Real-time data" refers to data that is obtained instantly from sensors, surveillance cameras, etc. and processed without delay.

[0879] "Workers" refers to workers who work within a logistics center.

[0880] An "automatic transport robot" is a robot that performs automated transport tasks within a logistics center.

[0881] "Providing instructions" refers to the act of communicating the work content and response methods to workers and automatic transport robots based on the analysis results.

[0882] An "abnormal condition" is a deviation from normal operating conditions that may affect safety or efficiency.

[0883] "Emergency response" refers to immediate response measures taken in the event of an abnormality or emergency.

[0884] The following describes in detail a system for implementing an application example of the present invention.

[0885] System Configuration

[0886] This system is designed to support efficient and safe operations in logistics centers and primarily consists of a server, sensors, surveillance cameras, terminals (smartphones or tablets) used by workers, and an automated transport robot.

[0887] Hardware and Software Use

[0888] Hardware:

[0889] Various sensors (temperature, humidity, location information, etc.)

[0890] surveillance cameras

[0891] Smartphone or tablet

[0892] Automatic transport robot

[0893] software:

[0894] Python for Data Analysis (Flask Framework)

[0895] Machine learning models (scikit-learn and TensorFlow)

[0896] WebSocket (JavaScript) for real-time data acquisition

[0897] Data processing and calculation

[0898] The server receives real-time data from various sensors in the logistics center via WebSocket. The received data is analyzed using a Flask-based API. A pre-trained machine learning model (scikit-learn or TensorFlow) is used for the analysis, and this model detects risk factors such as temperature rises or abnormal machine operation.

[0899] Operations and Examples

[0900] Operation steps:

[0901] 1. Real-time data acquisition: Collect data such as temperature, humidity, and location information from sensors within the logistics center.

[0902] 2. Sending data: The collected data is sent to the server via WebSocket.

[0903] 3. Data analysis: The server's machine learning model analyzes the data and extracts risk factors.

[0904] 4. Providing instructions: Based on the analysis results, appropriate instructions are provided in voice and text format to the worker's device and the automatic transport robot.

[0905] 5. Response to abnormalities: If an abnormality is detected, an emergency response will be immediately implemented and the system will take autopilot.

[0906] Examples:

[0907] For example, if a temperature sensor detects an abnormal temperature rise in an area of ​​a logistics center, this data is sent to a server in real time. The server's machine learning model uses this information to determine that there is a high risk of fire. The system immediately provides a voice instruction to workers saying, "Abnormal temperature rise detected in Area B. Please check immediately," and instructs an automated transport robot to move from the area to a safe location.

[0908] Example prompt sentence:

[0909] Design an AI system to collect and analyze real-time data from various sensors in a logistics center. Based on the analysis results, provide instructions to workers and automated transport robots, and automatically respond in the event of an abnormality.

[0910] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0911] Step 1:

[0912] The server collects data in real time from various sensors in the logistics center. As input, it receives data such as temperature, humidity, and location information from each sensor, and as output, it stores this data in its internal storage.

[0913] Specifically, the system obtains the value measured by the temperature sensor (e.g., 25 degrees) and the coordinates obtained by the position sensor (e.g., 35 degrees north latitude, 139 degrees east longitude).

[0914] Step 2:

[0915] The server collects video data from the surveillance cameras and audio data from the microphones. As input, it receives video frames from the cameras and audio data from the microphones, and as output, it passes these data to the video analysis and audio analysis modules.

[0916] Specifically, it captures camera video frames (e.g., JPEG images with a resolution of 1920x1080) and records audio data (e.g., conversations between controllers and workers).

[0917] Step 3:

[0918] The server inputs the collected data into a machine learning model for analysis. Data from sensors and surveillance cameras is received as input, and risk factors are extracted as analysis results as output. Specific data processing, for example, involves analyzing temperature data over time to detect abnormal temperature increases.

[0919] Specific actions include identifying abnormal patterns (e.g., a temperature spike lasting 10 minutes) and reporting them as a risk.

[0920] Step 4:

[0921] The server provides instructions to workers and automated transport robots based on the analysis results. It receives the risk factors of the analysis results as input, generates instructions in text and voice format as output, and sends them to the workers' terminals and the robots.

[0922] Specifically, the system sends a voice message to the worker's smartphone saying, "An abnormal temperature rise has been detected in Area B. Please check immediately," and instructs the robot to move to a safe location.

[0923] Step 5:

[0924] The server automatically executes emergency response when an anomaly is detected. It receives anomaly detection alerts as input and executes specific response measures as output.

[0925] Specifically, if it is determined that there is a high risk of fire, the system will immediately contact the fire department and move the automated transport robot to a safe location.

[0926] Step 6:

[0927] The server continuously trains the machine learning model using data collected in real time, taking newly collected data as input and improving the model's performance as output.

[0928] Specifically, the system uses previously collected data on normal and abnormal patterns as training data and updates the model to improve analysis accuracy.

[0929] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0930] ---

[0931] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[0932] System Configuration

[0933] 1. Real-time data acquisition

[0934] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[0935] 2. Analysis of video and audio data

[0936] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[0937] 3. User Emotion Recognition

[0938] The server provides the pilot's voice and biological information to the emotion engine, which analyzes the user's (pilot's) emotions. The emotion engine recognizes the pilot's emotional state from voice tone and physiological data (heart rate, skin potential response, etc.).

[0939] 4. Data analysis using AI models

[0940] The server inputs the flight data, video data, audio data, and emotion data into the AI ​​model for analysis. The AI ​​model integrates this data and extracts risk factors related to flight safety.

[0941] 5. Providing instructions to pilots

[0942] The server then uses the analysis results to provide instructions to the pilot in text and voice format. Using the results of the emotion engine, the server generates appropriate instructions based on the pilot's emotional state. For example, if the pilot is under high stress, the server provides concise and clear instructions, adding reassuring words.

[0943] 6. Emergency Detection

[0944] The server uses the analysis data to detect abnormal situations and emergencies in real time. It also takes into account data from the emotion engine to more accurately detect emergencies. For example, if a pilot's heart rate suddenly rises or their emotional state suddenly changes, it will notify them of that condition as a warning.

[0945] 7. Activating the autopilot system

[0946] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[0947] 8. Continuous learning

[0948] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[0949] ---

[0950] Processing flow and specific examples

[0951] Real-time data acquisition

[0952] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[0953] Video and audio data analysis

[0954] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[0955] User Emotion Recognition

[0956] The server analyzes the pilot's emotional state using an emotion engine. Based on collected physiological data such as voice tone and heart rate, it determines whether the pilot is under stress. For example, if the heart rate is abnormally high and the voice tone is tense, it determines that the pilot is under high stress.

[0957] Data analysis using AI models

[0958] The server inputs all acquired data into an AI model to extract safety-related risk factors. For example, the analysis results show that a sudden change in weather is predicted.

[0959] Providing instructions to pilots

[0960] Based on the analysis results, the server provides instructions to the pilot via text display and voice notification. The emotion engine's analysis results are reflected in the instructions generated based on the pilot's emotional state. For example, in a high-stress situation, additional instructions such as "Remain calm and maintain your current altitude" are given.

[0961] Emergency Detection

[0962] The server analyzes flight data to detect emergencies such as engine malfunctions or sudden weather changes. It also takes into account data from the emotion engine, and if an abnormal physiological state of the pilot is observed, it uses that information to issue a highly accurate warning. For example, if the engine temperature is abnormally high and the pilot's heart rate is also rising sharply, this will be detected as an emergency.

[0963] Autopilot system activation

[0964] If the server detects an emergency, it immediately activates the autopilot system and instructs the appropriate emergency response. For example, in the event of an engine malfunction, it will instruct the aircraft to make an emergency landing at an appropriate airport, and the autopilot system will fly that route autonomously.

[0965] Continuous learning

[0966] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns, for example, by improving the system's response algorithms based on newly observed emergency patterns.

[0967] The above are the processing steps of the system of the present invention, including specific examples. By combining it with an emotion engine, the safety of aircraft can be further improved and the mental burden on pilots can be effectively reduced.

[0968] The processing flow will be explained below.

[0969] Step 1:

[0970] The server receives real-time flight data from various sensors on the aircraft, including altitude, speed, geographic coordinates, and engine status. For example, the flight altitude is 30,000 feet, the speed is 500 knots, the geographic coordinates are 35 degrees north, 140 degrees east, and the engine status is normal.

[0971] Step 2:

[0972] The server receives camera footage from the airport. The video data is analyzed using image recognition technology to detect obstacles and abnormalities on the runway. For example, if a bird is detected on the runway through video analysis, it will identify it and issue a warning.

[0973] Step 3:

[0974] The server receives voice data from radio communications between the controller and the pilot and analyzes it using voice recognition technology. For example, it checks whether the controller's instructions, such as "Maintain current altitude," were properly conveyed to the pilot.

[0975] Step 4:

[0976] The server provides the pilot's voice, heart rate, and other physiological information to the emotion engine, which analyzes the user's (pilot's) emotional state. The emotion engine recognizes the pilot's stress and tension from the tone of their voice and heart rate. For example, if their heart rate is higher than normal and their voice tone is rising, it determines that the pilot is nervous.

[0977] Step 5:

[0978] The server inputs flight data, video data, audio data, and emotion data into the AI ​​model, and analyzes risk factors related to flight safety. For example, sudden weather changes and close proximity to other aircraft are extracted as risk factors.

[0979] Step 6:

[0980] The server generates appropriate instructions for the pilot based on the analysis results. It also reflects the results of the emotion engine and generates instructions that correspond to the pilot's emotional state. For example, it may say, "Please relax. You are instructed to maintain your current altitude."

[0981] Step 7:

[0982] The server detects emergencies from the analyzed data, including engine abnormalities, sudden weather changes, close encounters with other aircraft, etc. For example, if the engine temperature rises sharply and the pilot's heart rate also increases, this will be detected as an emergency.

[0983] Step 8:

[0984] If an emergency is detected, the server immediately activates the autopilot system and instructs the necessary emergency response. For example, in the event of an engine malfunction, the autopilot system can set up an emergency landing route to the nearest safe airport and fly that route autonomously.

[0985] Step 9:

[0986] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, it can learn new emergency patterns and improve prediction accuracy for the next flight.

[0987] The above are the detailed processing steps of the system of the present invention that combines the emotion engine. This is expected to further improve aircraft safety and effectively reduce the mental burden on pilots.

[0988] Example 2

[0989] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] Current aircraft systems face the problem of not being able to ensure flight safety or sufficiently reduce the pilot's mental burden by simply analyzing flight, video, and audio information. Failure to detect emergencies and provide appropriate instructions to pilots in real time increases the risk of serious accidents. Furthermore, there is no way to provide instructions that take into account the pilot's emotional state and stress level, which does not reduce the pilot's mental burden. A new system is needed to solve these issues and achieve safer and more efficient flight.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0992] In this invention, the server includes means for acquiring flight information, means for analyzing video and audio information, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight information, means for analyzing the emotional state of the pilot from physiological information and audio information, and means for generating appropriate aircraft operation instructions using the emotional state analysis results, thereby improving flight safety and reducing the mental burden on the pilot.

[0993] "Flight information" refers to real-time data such as an aircraft's altitude, speed, geographic coordinates, and engine status.

[0994] "Video and audio information" refers to digital video and audio data obtained from airport surveillance camera footage and radio communications with air traffic controllers.

[0995] "Analysis results" refers to the data obtained by processing flight information, video, and audio information, and specifically includes the detection of erroneous instructions and the extraction of risk factors.

[0996] "Aircraft operation instructions" refer to operational guidance and action instructions for the pilot that are generated based on the analysis results.

[0997] An "emergency" refers to a situation that threatens the safety of an aircraft, such as an engine malfunction, a sudden change in weather, or a physiological abnormality in the pilot.

[0998] An "autopilot system" is a system that operates an aircraft autonomously without pilot intervention in the event of an emergency.

[0999] "Machine learning model" refers to a statistical model that uses algorithms to accumulate knowledge and continuously learn, and includes models that make predictions and decisions based on flight information.

[1000] "Physiological information" refers to data that indicates the pilot's physical condition, such as their heart rate and skin potential response.

[1001] "Emotional state" refers to the pilot's psychological state as analyzed from their vocal tone and physiological information.

[1002] "Emotional state analysis results" refers to data on the pilot's emotional state analyzed by the emotion engine, including stress levels and tension.

[1003] "Appropriate aircraft operation instructions" refers to operational guidance and behavioral instructions that correspond to the pilot's mental condition and are generated based on the results of emotional state analysis.

[1004] MODE FOR CARRYING OUT THE INVENTION

[1005] An embodiment of the system of the present invention is described in detail below.

[1006] This system aims to improve aircraft safety by collecting and analyzing aircraft flight data, video and audio data, and physiological information. In particular, it analyzes the pilot's emotional state and generates appropriate instructions, thereby reducing the pilot's mental burden.

[1007] The server receives real-time flight data from the plane's various sensors. This includes the plane's altitude, speed, geographic coordinates, and engine status. For example, altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east. This data is retrieved from the aircraft's central system using a dedicated protocol.

[1008] The server collects video and audio data from airport surveillance cameras and wireless communications with air traffic controllers. Video data is acquired as streaming from the surveillance cameras, and audio data is received in digital audio format. Specifically, the system uses video analysis algorithms to detect foreign objects on the runway and voice recognition technology to analyze air traffic controllers' instructions in real time to check for erroneous instructions.

[1009] The server provides the pilot's voice and biometric information to the emotion engine, which analyzes their emotional state. The emotion engine analyzes their voice tone, heart rate, skin potential response, and other factors to recognize their emotional state. Data is collected from the audio microphone and biometric sensors and processed in real time. For example, if the pilot's heart rate is 30% higher than normal and their voice tone sounds tense, it is determined that the pilot is experiencing high levels of stress.

[1010] All data collected by the server (flight data, video data, audio data, emotion data) is input into the AI ​​model for analysis. The AI ​​model is composed of multiple neural networks and integrates and analyzes data. For example, sudden changes in weather are extracted as analysis results and reported as risk factors.

[1011] Based on the analysis results, the server provides instructions to the pilot in text and voice format. The instructions are generated based on the results of the emotion engine. For example, a pilot in a state of high stress may be given voice instructions such as "Remain calm and maintain your current altitude," which are also displayed on the screen in text format.

[1012] The server analyzes flight data in real time to detect abnormalities and emergencies. Emergencies include engine malfunctions, sudden weather changes, and abnormal pilot physiological states. For example, if the engine temperature suddenly rises and the pilot's heart rate also rises sharply, this will be detected as an emergency and an alert will be sent immediately.

[1013] When the server detects an emergency, it immediately activates the autopilot system, which executes emergency responses based on pre-programmed scenarios. For example, if an engine malfunction is detected, the autopilot system will execute an emergency landing route to the nearest airport.

[1014] The server continuously collects flight data and periodically updates the AI ​​model. This update process is performed offline, and new data sets are used to improve the model's performance. For example, new emergency scenarios can be learned and the system can adapt to them.

[1015] Prompt Sentence Examples

[1016] "The pilot's heart rate has been consistently 30% higher than normal. Please provide a specific example of what instructions should be given to the pilot if this condition is detected."

[1017] The above is an embodiment of the present invention, which can improve the safety of aircraft and effectively reduce the mental burden on pilots.

[1018] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1019] Step 1:

[1020] The server obtains flight data in real time from the aircraft's sensors. Specifically, data such as the aircraft's altitude, speed, geographic coordinates, and engine status are obtained from the aircraft's central system using a dedicated protocol. The input data is raw data from the sensors, which is collected in real time and stored in a database. As an output, various flight data are stored in the database in an organized manner.

[1021] Step 2:

[1022] The server acquires video data from surveillance cameras within the airport and collects audio data from radio communications with air traffic controllers. The video data is received in streaming format, and the audio data is received in digital audio format. The video data is run through an image recognition algorithm to detect foreign objects on the runway, and the audio data is run through voice recognition technology to check for erroneous instructions. The output is compiled into a report containing the foreign object detection results and audio analysis results.

[1023] Step 3:

[1024] The server collects the pilot's voice and physiological information and provides it to the emotion engine. Voice is captured through a microphone, and physiological data is acquired through a heart rate sensor and skin potential sensor. The emotion engine analyzes this data and recognizes the pilot's emotional state. The input is the voice and physiological data collected from the pilot, and the output is the pilot's emotional state (e.g., high stress, relaxed).

[1025] Step 4:

[1026] The flight data, video data, audio data, and emotional data acquired by the server are input into the AI ​​model. The AI ​​model includes a neural network that integrates and analyzes this data. The input data is initial data from various sensors, and the output is a report containing the integrated analysis results (e.g., sudden weather changes and risk factors). Specifically, the model performs functions such as detecting anomalies in flight data, detecting foreign objects in video data, and detecting erroneous instructions in audio data.

[1027] Step 5:

[1028] Based on the analysis results, the server provides instructions to the pilot in text and voice format. Instructions are generated based on the results of the emotion engine, and the pilot is given appropriate operational guidance. Analysis results and emotion data are used as input data, and the output is voice instructions (e.g., "Please remain calm and maintain your current altitude") and text instructions displayed on the pilot's display.

[1029] Step 6:

[1030] The server analyzes flight data and emotional data in real time to detect emergencies. If an engine abnormality, a sudden change in weather, or a pilot's physiological abnormality is detected, an emergency alert is issued immediately. The input is real-time flight data and emotional data, and the output is a warning message about the emergency and detailed information.

[1031] Step 7:

[1032] When the server detects an emergency, it activates the autopilot system. The autopilot system then takes appropriate action based on pre-programmed emergency response scenarios. For example, if an engine malfunction is detected, an emergency landing route to the nearest airport is automatically set and the flight is carried out along that route. The input is the emergency detection information, and the output is the activation of the autopilot system and a log of its actions.

[1033] Step 8:

[1034] The server continuously accumulates collected flight data and uses it to improve the performance of the AI ​​model. The newly acquired data is used to retrain the model, enabling more accurate analysis. The input is continuously collected flight data, and the output is an updated AI model and its performance evaluation results. For example, it is possible to learn new patterns of abnormal situations and incorporate appropriate countermeasures into the system.

[1035] (Application example 2)

[1036] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1037] In modern manufacturing, improving the efficiency and safety of factory operations are important challenges. In particular, there is a need to monitor the work environment and worker status in real time and respond quickly and appropriately when an abnormality occurs. However, existing systems lack the technology to integrate and analyze multiple data sources and automatically generate work instructions, leaving room for improvement. Therefore, there is a need for a system that can efficiently integrate and analyze real-time data and provide work instructions, including emergency responses.

[1038] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data and environmental data, means for acquiring sensor data within the factory, means for analyzing environmental data and physiological data, means for generating work instructions based on the analysis results, means for autonomously responding to emergencies based on the results of the analysis means, and means for updating the generating AI model and optimizing the instruction provision. This enables efficient work management and improved safety within the factory.

[1039] "Flight data" is data including aircraft altitude, speed, geographic coordinates, and engine status.

[1040] "Video and audio data" refers to data that includes video and audio from a surveillance camera.

[1041] "Aircraft operation instructions" are instructions regarding flight operations sent from the server to the pilot.

[1042] "Emergency" means an abnormal or dangerous condition in aircraft operations or factory operations.

[1043] An "autopilot system" is a system that controls the automatic operation of an aircraft.

[1044] A "machine learning model" is an artificial intelligence model that is trained to perform analysis and predictions using collected data.

[1045] "Sensor data" is data obtained from various sensors within the factory (temperature, humidity, machine operating status, etc.).

[1046] "Environmental data" refers to data including environmental conditions such as temperature, humidity, noise, and vibration within a factory.

[1047] "Physiological data" refers to data that indicates the physiological state of a worker, such as heart rate and skin potential.

[1048] "Work instructions" are specific instructions for actions provided to workers based on the analysis results.

[1049] A "generative AI model" is an artificial intelligence model that generates new instructions and countermeasures by analyzing data.

[1050] "Autonomous emergency response" means that the system automatically takes action to deal with an emergency.

[1051] As an embodiment of the present invention, the system is described in detail below. This system collects and analyzes flight data, video and audio data, sensor data, environmental data, physiological data, etc. in real time, and issues appropriate instructions and emergency responses based on the analysis results.

[1052] System Configuration

[1053] 1. Real-time data acquisition

[1054] The server receives real-time flight data from various sensors on the plane, including altitude, speed, geographic coordinates, and engine status, as well as sensor data from the factory, such as temperature, humidity, and machine operation status.

[1055] 2. Analysis of video and audio data

[1056] The server acquires surveillance camera footage and environmental audio, and uses video recognition technology to extract important visual information and audio recognition technology to detect signs of abnormalities.

[1057] 3. Physiological Data Collection and Analysis

[1058] The server collects physiological data such as the worker's heart rate and provides it to the emotion engine to analyze the worker's stress level. This analysis determines the worker's workload and takes the necessary measures.

[1059] 4. Data analysis using AI models

[1060] The server inputs the acquired flight data, video and audio data, sensor data, and physiological data into the AI ​​model, which then integrates and analyzes this data to identify risk factors both inside and outside the factory, contributing to improved safety and efficiency.

[1061] 5. Instruction Generation and Delivery

[1062] Based on the analysis, the server generates appropriate instructions for an aircraft pilot or a factory worker, delivered in text or audio format and including specific advice based on the worker's emotional state.

[1063] 6. Emergency Detection and Autonomous Response

[1064] The server uses the analysis data to detect abnormalities and emergencies in real time, and activates the autopilot system or the factory's emergency response system as necessary, allowing for a quick and safe response.

[1065] 7. Continuous learning

[1066] The server continuously updates the machine learning model using the collected data to improve the performance of the entire system, allowing it to quickly incorporate new abnormal patterns.

[1067] Hardware and software used

[1068] The hardware used is as follows:

[1069] Sensors: Various sensors that collect environmental data within the factory in real time

[1070] Camera: A surveillance camera for capturing video data

[1071] Physiological data monitor: A monitor that acquires information such as the worker's heart rate

[1072] The software used is as follows:

[1073] Pandas: Real-time data acquisition in dataframe format

[1074] OpenCV: Camera image analysis

[1075] Librosa: Analysis of audio data

[1076] Scikit-learn: Building and analyzing machine learning models

[1077] Specific examples

[1078] For example, if a factory machine detects abnormal vibrations, the system will immediately analyze the vibration data and provide appropriate instructions to workers. If a worker's heart rate is abnormally high, the system will instruct them to stop work and take a break to protect their health.

[1079] Prompt Sentence Examples

[1080] Here are some example prompts to input to the generative AI model:

[1081] "Sensor data, camera footage, and audio data from within the factory are collected in real time, and workers' physiological data is analyzed to detect abnormalities. All data is analyzed using AI models, risk assessment is performed, and appropriate instructions are provided to workers."

[1082] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1083] Step 1:

[1084] The server obtains flight data such as altitude, speed, geographic coordinates, and engine status from various sensors on the aircraft in real time. In addition, it also collects sensor data such as temperature, humidity, and machine operation status inside the factory. The input is sensor data, and the output is collected data in data frame format. The data is managed using the Pandas library.

[1085] Step 2:

[1086] The server acquires surveillance camera footage and factory audio data, and uses video and audio recognition technologies to detect abnormalities. The video data is analyzed using the OpenCV library, and the audio data is analyzed using the Librosa library. The input is the camera footage and audio data, and the output is a flag indicating whether or not an abnormality is present.

[1087] Step 3:

[1088] The server collects physiological data such as the worker's heart rate in real time and inputs it into the emotion engine. The emotion engine analyzes the biosignals and evaluates the worker's stress level. The input is physiological data and the output is stress level. Specifically, the emotional state is estimated from the heart rate and skin potential response.

[1089] Step 4:

[1090] The server inputs all collected flight data, video and audio data, sensor data, and physiological data into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors inside and outside the factory. The input is all collected data, and the output is the risk assessment results. The machine learning model is trained using Scikit-learn.

[1091] Step 5:

[1092] The server generates and notifies appropriate instructions to aircraft pilots and factory workers based on the analysis results. Instructions are provided in text and voice format. The input is the analysis result of the AI ​​model, and the output is the instruction content. Instructions are generated taking into account the results of the emotion engine.

[1093] Step 6:

[1094] The server detects abnormalities and emergencies in real time from the analysis data and activates the autopilot system or the factory's emergency response system as necessary. The input is real-time analysis data, and the output is the emergency state and the corresponding action. Specifically, when an abnormality is detected, the response system is immediately activated.

[1095] Step 7:

[1096] The server uses the collected data to continuously update the machine learning model, improving overall system performance. This allows new abnormal patterns to be quickly incorporated. The input is past collected data and new data, and the output is an updated AI model. New algorithms and parameter adjustments are also applied to the data as it is learned.

[1097] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1098] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1099] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1100] [Fourth embodiment]

[1101] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1102] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1103] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1104] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1105] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1106] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1107] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1108] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1109] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1110] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1111] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1112] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1113] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1114] ---

[1115] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[1116] System Configuration

[1117] 1. Real-time data acquisition

[1118] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[1119] 2. Analysis of video and audio data

[1120] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[1121] 3. Data analysis using AI models

[1122] The server uses an AI model (machine learning algorithm) to analyze the flight data and analyzed video and audio data, thereby extracting risk factors related to flight safety and supporting appropriate decisions based on the analysis results.

[1123] 4. Providing instructions to pilots

[1124] The server then uses the analysis to provide instructions to the pilot in text and audio format, such as "maintain altitude" and "follow the designated route."

[1125] 5. Emergency Detection

[1126] The server uses the analyzed data to detect abnormalities and emergencies in real time, including engine malfunctions, sudden weather changes, and close encounters with other aircraft.

[1127] 6. Activating the autopilot system

[1128] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[1129] 7. Continuous learning

[1130] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[1131] ---

[1132] Processing flow and specific examples

[1133] Real-time data acquisition

[1134] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[1135] Video and audio data analysis

[1136] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[1137] Data analysis using AI models

[1138] The server inputs all acquired data into an AI model and extracts safety-related risk factors. For example, the analysis results show that sudden weather changes are predicted.

[1139] Providing instructions to pilots

[1140] Based on the analysis results, the server provides instructions to the pilot via text and voice notification, such as "Please change to a new route" to respond to weather changes.

[1141] Emergency Detection

[1142] The server analyzes flight data and detects emergencies such as engine malfunctions or sudden weather changes. For example, if an abnormally high engine temperature is detected, a notification will be sent.

[1143] Autopilot system activation

[1144] If an emergency is detected, the server will activate the autopilot system and take appropriate action, such as specifying an emergency landing route to an appropriate airport in the event of an engine malfunction.

[1145] Continuous learning

[1146] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns.

[1147] ---

[1148] As described above, the system of the present invention can improve aircraft safety and reduce the burden on pilots. This system utilizes advanced data analysis technology and machine learning to set a new standard in the aviation industry.

[1149] The processing flow will be explained below.

[1150] Step 1:

[1151] The server receives real-time flight data from various sensors on the plane. This data includes altitude, speed, geographic coordinates, and engine status. For example, if the plane is at an altitude of 10,000 feet and traveling at 550 knots, that data is sent to the server.

[1152] Step 2:

[1153] The server acquires video data from the airport. It collects real-time camera footage from within the airport and uses image recognition technology to analyze the runway conditions and the presence of obstacles in the surrounding area. For example, it can detect foreign objects on the runway from the camera footage.

[1154] Step 3:

[1155] The server acquires the voice data of the communication between the controller and the pilot and analyzes it using voice recognition technology. This detects erroneous instructions and mishearing by the pilot. For example, if the controller instructs "Turn right" but the pilot mistakenly interprets it as "Turn left," the system will detect the misunderstanding.

[1156] Step 4:

[1157] The server inputs the flight data, video data, and audio data it acquires into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors related to flight safety. For example, if a sudden change in weather is predicted, that information is extracted as a risk factor.

[1158] Step 5:

[1159] Based on the analysis results, the server generates appropriate instructions for the pilot. The generated instructions are displayed in text format on the cockpit display and also communicated to the pilot in voice format. For example, the instruction "Maintain altitude" is displayed and the voice also communicates "Maintain altitude."

[1160] Step 6:

[1161] The server detects emergencies from the analysis data, such as engine abnormalities, sudden weather changes, or close proximity to other aircraft. For example, if the engine temperature is abnormally high, it will be detected as an emergency.

[1162] Step 7:

[1163] When an emergency is detected, the server activates the autopilot system, which then executes the appropriate emergency response procedures. For example, in the event of an engine failure, the autopilot system calculates an emergency landing route to the nearest airport and autonomously directs the plane along that route.

[1164] Step 8:

[1165] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, new flight data can be used to refine the fuel efficiency optimization algorithm for the next flight.

[1166] These are the processing steps of the system of the present invention, including specific examples, which improves aircraft safety and reduces pilot workload.

[1167] Example 1

[1168] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1169] Improving safety and reducing pilot workload are key issues in modern aviation. In particular, failure to properly detect anomalies and respond to emergencies in real time poses a risk of serious accidents. Systems that efficiently analyze flight data and provide accurate flight instructions to pilots are also needed. Furthermore, continuous system performance improvement through machine learning is also necessary.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1171] In this invention, the server includes a means for acquiring flight data, a means for analyzing surveillance camera and wireless communication data, a means for generating aircraft operation instructions based on the analysis results, a means for detecting an emergency, a means for activating an autopilot system when the emergency is detected, and a means for continuously training a machine learning model using the flight data and analysis data. This enables abnormalities to be detected in real time and appropriate countermeasures to be implemented promptly. Furthermore, continuous learning improves the accuracy of the system, contributing to aircraft safety and reducing the burden on pilots.

[1172] "Flight data" is data that includes information such as aircraft altitude, speed, geographic coordinates, and engine status.

[1173] A "surveillance camera" is a device that captures images for the purpose of monitoring airports and aircraft.

[1174] "Radio communication data" refers to data of voice communications between air traffic controllers and pilots.

[1175] The "analysis results" are the analysis results obtained after analyzing the acquired data.

[1176] "Aircraft operation instructions" refers to specific instructions regarding the piloting and operation of an aircraft based on the analysis results.

[1177] An "emergency situation" is an abnormal condition that poses a significant risk to the safe operation of an aircraft.

[1178] An "autopilot system" is a system for automatically operating an aircraft.

[1179] A "machine learning model" refers to an algorithm or system that automatically learns from data and makes predictions and classifications.

[1180] "Continuous learning" is the process of constantly updating a model's learning with new data to improve its accuracy and performance.

[1181] MODE FOR CARRYING OUT THE INVENTION

[1182] The following describes in detail the mode for carrying out the present invention. The purpose of this system is to improve aircraft safety and reduce the burden on pilots. The hardware and software used, as well as the methods of data processing and calculation, are explained below.

[1183] System Configuration

[1184] Real-time data acquisition

[1185] The server obtains flight data in real time from various sensors on the aircraft. This flight data includes altitude, speed, geographic coordinates, and engine status. The server collects data using the aircraft's altimeter, speedometer, GPS, and engine monitoring system, and stores it in a database (e.g., MySQL) in real time.

[1186] As a specific example, the server obtains information in seconds about an airplane in flight, such as its altitude, speed, latitude, and longitude, which are 3,000 meters, 900 km / h, 35 degrees north, and 139 degrees east, and stores this information in a database.

[1187] Video and audio data analysis

[1188] The server collects and analyzes airport surveillance camera footage and radio communication data between controllers and pilots. Specifically, the server acquires surveillance camera footage and uses image recognition software (e.g., OpenCV) to extract important visual information. It also analyzes controller-pilot communications using voice recognition technology (e.g., Google Speech-to-Text API) to detect miscommunications and mishearing.

[1189] Specifically, the server detects moving objects on the runway from surveillance camera footage and uses voice recognition to confirm that the controller's instruction is to "climb 500 feet."

[1190] Data analysis using AI models

[1191] The server inputs the acquired data into a machine learning model for analysis. The server inputs flight data and video / audio analysis data acquired from the database into an AI model (e.g., TensorFlow), analyzes the data, and extracts potential risk factors. The analysis results are classified by risk type and sent to the server's internal evaluation system.

[1192] As a specific example, the server obtains predictions of sudden weather changes from an AI model and reflects this information in the risk assessment system.

[1193] Providing instructions to pilots

[1194] The server provides text and voice instructions to the pilot based on the analysis results. For example, the server checks the analysis results, generates text instructions, and displays them on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. A specific example of such an instruction would be "Please change to a new route" displayed on the screen and also given as a voice instruction.

[1195] Emergency Detection

[1196] The server detects emergencies based on real-time data analysis results. The server periodically monitors the analysis results and runs algorithms to detect abnormal patterns. If an emergency is detected, it immediately generates an alert and notifies relevant parties. Detailed information about the emergency is provided to the pilot.

[1197] As a specific example, the server detects an abnormal rise in engine temperature and notifies the pilot with an alert on the screen and via voice.

[1198] Autopilot system activation

[1199] When the server detects an emergency, it activates the autopilot system. When the server detects an emergency, it issues a command to activate the autopilot mode. It also sends a pre-defined emergency response route to the autopilot system and executes it. The autopilot system then initiates the designated emergency response.

[1200] As a specific example, if an engine malfunction is detected, the server will activate the autopilot system and route the aircraft to the nearest suitable airport for an emergency landing.

[1201] Continuous learning

[1202] The server continuously trains the AI ​​model using the acquired data, adds newly acquired flight data to the machine learning dataset, and periodically retrains the machine learning model to improve accuracy and performance, and deploys the updated model to the system.

[1203] For example, new flight data can be used to retrain AI models, improving the accuracy of anomaly detection.

[1204] Prompt Sentence Examples

[1205] "Please explain the specific processing flow of the system that acquires real-time data from the aircraft, analyzes video and audio to detect abnormalities, and issues instructions to the pilot. Please also explain the specific operation of each step."

[1206] As described above, the system of the present invention aims to improve aircraft safety and reduce pilot workload by utilizing advanced data analysis technology and machine learning, and this system can set a new standard in the aviation industry.

[1207] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1208] Program processing flow

[1209] Step 1:

[1210] The server acquires flight data in real time from various sensors on the aircraft. The input for this step is data from the altimeter, speedometer, GPS, and engine monitoring system installed on the aircraft. The server receives this data and stores it in a database (e.g., MySQL) in real time. Specifically, the server acquires data every second, such as an altitude of 3,000 meters, a speed of 900 km / h, latitude 35 degrees north, and longitude 139 degrees east, and stores it in the database.

[1211] Step 2:

[1212] The server collects and analyzes airport surveillance camera footage and radio communication data between the controller and pilot. The input for this step is live video data from the surveillance cameras and audio data from radio communication. The server uses OpenCV to analyze the video data and detect moving objects and anomalies. It also uses the Google Speech-to-Text API to convert audio data into text and detect erroneous instructions or mishearing. Specifically, the server detects foreign objects on the runway from surveillance camera footage and uses voice recognition to confirm the controller's instruction to "climb to an altitude of 500 feet."

[1213] Step 3:

[1214] The server inputs the acquired flight data and analyzed video and audio data into the AI ​​model for analysis. The inputs for this step are the flight data, video data, and audio data acquired in the previous step. The server uses TensorFlow to analyze this data and extract potential risk factors. The analysis results are classified by risk type and sent to the built-in evaluation system. Specifically, the server receives a result from the AI ​​model indicating that a sudden weather change is predicted, and reflects this information in the risk evaluation system.

[1215] Step 4:

[1216] Based on the analysis results, the server provides text and voice instructions to the pilot. The input for this step is the AI ​​analysis results. The server checks the analysis results, generates instructions, and displays them in text on the pilot's display. It also generates voice instructions and sends them to the pilot's headset. Specifically, the instruction "Please change to a new route" is displayed on the screen and is also given voice instructions.

[1217] Step 5:

[1218] The server detects emergencies based on the results of real-time data analysis. The input to this step is the analysis results. The server periodically monitors the analysis results and runs an algorithm to detect abnormal patterns. If an emergency is detected, an alert is immediately generated and notified to the pilot. Specifically, the server detects an abnormal rise in engine temperature and notifies the pilot of the alert on the screen and by voice.

[1219] Step 6:

[1220] If the server detects an emergency, it activates the autopilot system. The input to this step is the emergency detection result. The server issues a command to activate the autopilot mode and sends a pre-set emergency response route to the autopilot system to execute. Specifically, if an engine abnormality is detected, the autopilot system is activated and an emergency landing route to the nearest appropriate airport is specified.

[1221] Step 7:

[1222] The server continuously trains the AI ​​model using the acquired data. The input for this step is newly acquired flight data. The server adds this to the machine learning dataset and periodically retrains the machine learning model to improve accuracy and performance. The updated model is then deployed to the system. Specifically, the AI ​​model is retrained using new flight data, improving the accuracy of anomaly detection.

[1223] Through the above processing steps, this system can improve aircraft safety and reduce the burden on pilots.

[1224] (Application example 1)

[1225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1226] Operations at conventional logistics centers depended heavily on human labor, posing safety and efficiency challenges. For example, when equipment failure, abnormal temperature rises, or operational errors occurred, it was difficult to immediately detect and respond to these risks. Furthermore, due to insufficient real-time data analysis and automation, there were frequent delays in response in situations where appropriate decisions needed to be made immediately. Therefore, there was an urgent need to provide a system that would improve the safety and efficiency of logistics centers.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1228] In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data, means for acquiring various data within the logistics center in real time, means for providing instructions to workers and automated transport robots based on the analysis results, and means for automatically executing an emergency response in the event of an abnormality. This makes it possible to analyze real-time data from sensors and surveillance cameras in the logistics center and automate the provision of instructions to workers and automated transport robots and emergency responses.

[1229] "Flight data" refers to various data relating to the operation of an aircraft, including the aircraft's altitude, speed, geographic coordinates, engine status, etc.

[1230] "Video Data" means images or video information captured by surveillance cameras or other visual sensors.

[1231] "Voice data" means data containing communications between air traffic controllers and pilots and other voice information.

[1232] "Analysis results" refer to the results of analyzing the acquired data using AI models or machine learning algorithms.

[1233] "Aircraft operation instructions" refer to specific instructions regarding aircraft operation based on the analysis results, such as instructions to maintain altitude or change route.

[1234] An "emergency" is an abnormal condition, such as an engine malfunction or a sudden change in weather, that poses a significant risk to aircraft operation.

[1235] An "autopilot system" is an automated flight system that assists or substitutes for pilot operations under certain conditions, such as emergencies.

[1236] A "machine learning model" is an algorithm or program that continuously learns from collected data and improves the accuracy of analysis.

[1237] A "logistics center" is a facility that receives, stores, and ships goods.

[1238] "Real-time data" refers to data that is obtained instantly from sensors, surveillance cameras, etc. and processed without delay.

[1239] "Workers" refers to workers who work within a logistics center.

[1240] An "automatic transport robot" is a robot that performs automated transport tasks within a logistics center.

[1241] "Providing instructions" refers to the act of communicating the work content and response methods to workers and automatic transport robots based on the analysis results.

[1242] An "abnormal condition" is a deviation from normal operating conditions that may affect safety or efficiency.

[1243] "Emergency response" refers to immediate response measures taken in the event of an abnormality or emergency.

[1244] The following describes in detail a system for implementing an application example of the present invention.

[1245] System Configuration

[1246] This system is designed to support efficient and safe operations in logistics centers and primarily consists of a server, sensors, surveillance cameras, terminals (smartphones or tablets) used by workers, and an automated transport robot.

[1247] Hardware and Software Use

[1248] Hardware:

[1249] Various sensors (temperature, humidity, location information, etc.)

[1250] surveillance cameras

[1251] Smartphone or tablet

[1252] Automatic transport robot

[1253] software:

[1254] Python for Data Analysis (Flask Framework)

[1255] Machine learning models (scikit-learn and TensorFlow)

[1256] WebSocket (JavaScript) for real-time data acquisition

[1257] Data processing and calculation

[1258] The server receives real-time data from various sensors in the logistics center via WebSocket. The received data is analyzed using a Flask-based API. A pre-trained machine learning model (scikit-learn or TensorFlow) is used for the analysis, and this model detects risk factors such as temperature rises or abnormal machine operation.

[1259] Operations and Examples

[1260] Operation steps:

[1261] 1. Real-time data acquisition: Collect data such as temperature, humidity, and location information from sensors within the logistics center.

[1262] 2. Sending data: The collected data is sent to the server via WebSocket.

[1263] 3. Data analysis: The server's machine learning model analyzes the data and extracts risk factors.

[1264] 4. Providing instructions: Based on the analysis results, appropriate instructions are provided in voice and text format to the worker's device and the automatic transport robot.

[1265] 5. Response to abnormalities: If an abnormality is detected, an emergency response will be immediately implemented and the system will take autopilot.

[1266] Examples:

[1267] For example, if a temperature sensor detects an abnormal temperature rise in an area of ​​a logistics center, this data is sent to a server in real time. The server's machine learning model uses this information to determine that there is a high risk of fire. The system immediately provides a voice instruction to workers saying, "Abnormal temperature rise detected in Area B. Please check immediately," and instructs an automated transport robot to move from the area to a safe location.

[1268] Example prompt sentence:

[1269] Design an AI system to collect and analyze real-time data from various sensors in a logistics center. Based on the analysis results, provide instructions to workers and automated transport robots, and automatically respond in the event of an abnormality.

[1270] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1271] Step 1:

[1272] The server collects data in real time from various sensors in the logistics center. As input, it receives data such as temperature, humidity, and location information from each sensor, and as output, it stores this data in its internal storage.

[1273] Specifically, the system obtains the value measured by the temperature sensor (e.g., 25 degrees) and the coordinates obtained by the position sensor (e.g., 35 degrees north latitude, 139 degrees east longitude).

[1274] Step 2:

[1275] The server collects video data from the surveillance cameras and audio data from the microphones. As input, it receives video frames from the cameras and audio data from the microphones, and as output, it passes these data to the video analysis and audio analysis modules.

[1276] Specifically, it captures camera video frames (e.g., JPEG images with a resolution of 1920x1080) and records audio data (e.g., conversations between controllers and workers).

[1277] Step 3:

[1278] The server inputs the collected data into a machine learning model for analysis. Data from sensors and surveillance cameras is received as input, and risk factors are extracted as analysis results as output. Specific data processing, for example, involves analyzing temperature data over time to detect abnormal temperature increases.

[1279] Specific actions include identifying abnormal patterns (e.g., a temperature spike lasting 10 minutes) and reporting them as a risk.

[1280] Step 4:

[1281] The server provides instructions to workers and automated transport robots based on the analysis results. It receives the risk factors of the analysis results as input, generates instructions in text and voice format as output, and sends them to the workers' terminals and the robots.

[1282] Specifically, the system sends a voice message to the worker's smartphone saying, "An abnormal temperature rise has been detected in Area B. Please check immediately," and instructs the robot to move to a safe location.

[1283] Step 5:

[1284] The server automatically executes emergency response when an anomaly is detected. It receives anomaly detection alerts as input and executes specific response measures as output.

[1285] Specifically, if it is determined that there is a high risk of fire, the system will immediately contact the fire department and move the automated transport robot to a safe location.

[1286] Step 6:

[1287] The server continuously trains the machine learning model using data collected in real time, taking newly collected data as input and improving the model's performance as output.

[1288] Specifically, the system uses previously collected data on normal and abnormal patterns as training data and updates the model to improve analysis accuracy.

[1289] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1290] ---

[1291] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below as an embodiment thereof.

[1292] System Configuration

[1293] 1. Real-time data acquisition

[1294] The server receives real-time flight data from various sensors on the plane, including the plane's altitude, speed, geographic coordinates, and engine status.

[1295] 2. Analysis of video and audio data

[1296] The server collects and analyzes video and audio data from the airport. Specifically, it acquires camera footage from within the airport and uses image recognition technology to extract important visual information. It also acquires communication data between controllers and pilots and uses voice recognition technology to detect miscommunications and mishearing.

[1297] 3. User Emotion Recognition

[1298] The server provides the pilot's voice and biological information to the emotion engine, which analyzes the user's (pilot's) emotions. The emotion engine recognizes the pilot's emotional state from voice tone and physiological data (heart rate, skin potential response, etc.).

[1299] 4. Data analysis using AI models

[1300] The server inputs the flight data, video data, audio data, and emotion data into the AI ​​model for analysis. The AI ​​model integrates this data and extracts risk factors related to flight safety.

[1301] 5. Providing instructions to pilots

[1302] The server then uses the analysis results to provide instructions to the pilot in text and voice format. Using the results of the emotion engine, the server generates appropriate instructions based on the pilot's emotional state. For example, if the pilot is under high stress, the server provides concise and clear instructions, adding reassuring words.

[1303] 6. Emergency Detection

[1304] The server uses the analysis data to detect abnormal situations and emergencies in real time. It also takes into account data from the emotion engine to more accurately detect emergencies. For example, if a pilot's heart rate suddenly rises or their emotional state suddenly changes, it will notify them of that condition as a warning.

[1305] 7. Activating the autopilot system

[1306] If the server detects an emergency, it immediately activates the autopilot system and takes necessary emergency measures, reducing the burden on the pilot and enabling safe responses.

[1307] 8. Continuous learning

[1308] The server continuously collects flight data and updates the machine learning model, improving the overall system performance and providing more accurate analysis results.

[1309] ---

[1310] Processing flow and specific examples

[1311] Real-time data acquisition

[1312] The server collects real-time data from sensors during flight, including the plane's altitude, speed, coordinates, engine status, etc. For example, the information could be: altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east.

[1313] Video and audio data analysis

[1314] The server analyzes the airport's surveillance camera footage and radio communications with air traffic controllers. For example, video analysis can be used to detect foreign objects on the runway, and audio analysis can be used to detect erroneous instructions.

[1315] User Emotion Recognition

[1316] The server analyzes the pilot's emotional state using an emotion engine. Based on collected physiological data such as voice tone and heart rate, it determines whether the pilot is under stress. For example, if the heart rate is abnormally high and the voice tone is tense, it determines that the pilot is under high stress.

[1317] Data analysis using AI models

[1318] The server inputs all acquired data into an AI model to extract safety-related risk factors. For example, the analysis results show that a sudden change in weather is predicted.

[1319] Providing instructions to pilots

[1320] Based on the analysis results, the server provides instructions to the pilot via text display and voice notification. The emotion engine's analysis results are reflected in the instructions generated based on the pilot's emotional state. For example, in a high-stress situation, additional instructions such as "Remain calm and maintain your current altitude" are given.

[1321] Emergency Detection

[1322] The server analyzes flight data to detect emergencies such as engine malfunctions or sudden weather changes. It also takes into account data from the emotion engine, and if an abnormal physiological state of the pilot is observed, it uses that information to issue a highly accurate warning. For example, if the engine temperature is abnormally high and the pilot's heart rate is also rising sharply, this will be detected as an emergency.

[1323] Autopilot system activation

[1324] If the server detects an emergency, it immediately activates the autopilot system and instructs the appropriate emergency response. For example, in the event of an engine malfunction, it will instruct the aircraft to make an emergency landing at an appropriate airport, and the autopilot system will fly that route autonomously.

[1325] Continuous learning

[1326] The server continuously updates the AI ​​model using the collected data, allowing it to optimize fuel efficiency and recognize new abnormal patterns, for example, by improving the system's response algorithms based on newly observed emergency patterns.

[1327] The above are the processing steps of the system of the present invention, including specific examples. By combining it with an emotion engine, the safety of aircraft can be further improved and the mental burden on pilots can be effectively reduced.

[1328] The processing flow will be explained below.

[1329] Step 1:

[1330] The server receives real-time flight data from various sensors on the aircraft, including altitude, speed, geographic coordinates, and engine status. For example, the flight altitude is 30,000 feet, the speed is 500 knots, the geographic coordinates are 35 degrees north, 140 degrees east, and the engine status is normal.

[1331] Step 2:

[1332] The server receives camera footage from the airport. The video data is analyzed using image recognition technology to detect obstacles and abnormalities on the runway. For example, if a bird is detected on the runway through video analysis, it will identify it and issue a warning.

[1333] Step 3:

[1334] The server receives voice data from radio communications between the controller and the pilot and analyzes it using voice recognition technology. For example, it checks whether the controller's instructions, such as "Maintain current altitude," were properly conveyed to the pilot.

[1335] Step 4:

[1336] The server provides the pilot's voice, heart rate, and other physiological information to the emotion engine, which analyzes the user's (pilot's) emotional state. The emotion engine recognizes the pilot's stress and tension from the tone of their voice and heart rate. For example, if their heart rate is higher than normal and their voice tone is rising, it determines that the pilot is nervous.

[1337] Step 5:

[1338] The server inputs flight data, video data, audio data, and emotion data into the AI ​​model, and analyzes risk factors related to flight safety. For example, sudden weather changes and close proximity to other aircraft are extracted as risk factors.

[1339] Step 6:

[1340] The server generates appropriate instructions for the pilot based on the analysis results. It also reflects the results of the emotion engine and generates instructions that correspond to the pilot's emotional state. For example, it may say, "Please relax. You are instructed to maintain your current altitude."

[1341] Step 7:

[1342] The server detects emergencies from the analyzed data, including engine abnormalities, sudden weather changes, close encounters with other aircraft, etc. For example, if the engine temperature rises sharply and the pilot's heart rate also increases, this will be detected as an emergency.

[1343] Step 8:

[1344] If an emergency is detected, the server immediately activates the autopilot system and instructs the necessary emergency response. For example, in the event of an engine malfunction, the autopilot system can set up an emergency landing route to the nearest safe airport and fly that route autonomously.

[1345] Step 9:

[1346] The server continuously collects flight data and updates the machine learning model, improving the system's analysis accuracy. For example, it can learn new emergency patterns and improve prediction accuracy for the next flight.

[1347] The above are the detailed processing steps of the system of the present invention that combines the emotion engine. This is expected to further improve aircraft safety and effectively reduce the mental burden on pilots.

[1348] Example 2

[1349] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1350] Current aircraft systems face the problem of not being able to ensure flight safety or sufficiently reduce the pilot's mental burden by simply analyzing flight, video, and audio information. Failure to detect emergencies and provide appropriate instructions to pilots in real time increases the risk of serious accidents. Furthermore, there is no way to provide instructions that take into account the pilot's emotional state and stress level, which does not reduce the pilot's mental burden. A new system is needed to solve these issues and achieve safer and more efficient flight.

[1351] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1352] In this invention, the server includes means for acquiring flight information, means for analyzing video and audio information, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight information, means for analyzing the emotional state of the pilot from physiological information and audio information, and means for generating appropriate aircraft operation instructions using the emotional state analysis results, thereby improving flight safety and reducing the mental burden on the pilot.

[1353] "Flight information" refers to real-time data such as an aircraft's altitude, speed, geographic coordinates, and engine status.

[1354] "Video and audio information" refers to digital video and audio data obtained from airport surveillance camera footage and radio communications with air traffic controllers.

[1355] "Analysis results" refers to the data obtained by processing flight information, video, and audio information, and specifically includes the detection of erroneous instructions and the extraction of risk factors.

[1356] "Aircraft operation instructions" refer to operational guidance and action instructions for the pilot that are generated based on the analysis results.

[1357] An "emergency" refers to a situation that threatens the safety of an aircraft, such as an engine malfunction, a sudden change in weather, or a physiological abnormality in the pilot.

[1358] An "autopilot system" is a system that operates an aircraft autonomously without pilot intervention in the event of an emergency.

[1359] "Machine learning model" refers to a statistical model that uses algorithms to accumulate knowledge and continuously learn, and includes models that make predictions and decisions based on flight information.

[1360] "Physiological information" refers to data that indicates the pilot's physical condition, such as their heart rate and skin potential response.

[1361] "Emotional state" refers to the pilot's psychological state as analyzed from their vocal tone and physiological information.

[1362] "Emotional state analysis results" refers to data on the pilot's emotional state analyzed by the emotion engine, including stress levels and tension.

[1363] "Appropriate aircraft operation instructions" refers to operational guidance and behavioral instructions that correspond to the pilot's mental condition and are generated based on the results of emotional state analysis.

[1364] MODE FOR CARRYING OUT THE INVENTION

[1365] An embodiment of the system of the present invention is described in detail below.

[1366] This system aims to improve aircraft safety by collecting and analyzing aircraft flight data, video and audio data, and physiological information. In particular, it analyzes the pilot's emotional state and generates appropriate instructions, thereby reducing the pilot's mental burden.

[1367] The server receives real-time flight data from the plane's various sensors. This includes the plane's altitude, speed, geographic coordinates, and engine status. For example, altitude 3,000 meters, speed 900 km / h, latitude 35 degrees north, longitude 139 degrees east. This data is retrieved from the aircraft's central system using a dedicated protocol.

[1368] The server collects video and audio data from airport surveillance cameras and wireless communications with air traffic controllers. Video data is acquired as streaming from the surveillance cameras, and audio data is received in digital audio format. Specifically, the system uses video analysis algorithms to detect foreign objects on the runway and voice recognition technology to analyze air traffic controllers' instructions in real time to check for erroneous instructions.

[1369] The server provides the pilot's voice and biometric information to the emotion engine, which analyzes their emotional state. The emotion engine analyzes their voice tone, heart rate, skin potential response, and other factors to recognize their emotional state. Data is collected from the audio microphone and biometric sensors and processed in real time. For example, if the pilot's heart rate is 30% higher than normal and their voice tone sounds tense, it is determined that the pilot is experiencing high levels of stress.

[1370] All data collected by the server (flight data, video data, audio data, emotion data) is input into the AI ​​model for analysis. The AI ​​model is composed of multiple neural networks and integrates and analyzes data. For example, sudden changes in weather are extracted as analysis results and reported as risk factors.

[1371] Based on the analysis results, the server provides instructions to the pilot in text and voice format. The instructions are generated based on the results of the emotion engine. For example, a pilot in a state of high stress may be given voice instructions such as "Remain calm and maintain your current altitude," which are also displayed on the screen in text format.

[1372] The server analyzes flight data in real time to detect abnormalities and emergencies. Emergencies include engine malfunctions, sudden weather changes, and abnormal pilot physiological states. For example, if the engine temperature suddenly rises and the pilot's heart rate also rises sharply, this will be detected as an emergency and an alert will be sent immediately.

[1373] When the server detects an emergency, it immediately activates the autopilot system, which executes emergency responses based on pre-programmed scenarios. For example, if an engine malfunction is detected, the autopilot system will execute an emergency landing route to the nearest airport.

[1374] The server continuously collects flight data and periodically updates the AI ​​model. This update process is performed offline, and new data sets are used to improve the model's performance. For example, new emergency scenarios can be learned and the system can adapt to them.

[1375] Prompt Sentence Examples

[1376] "The pilot's heart rate has been consistently 30% higher than normal. Please provide a specific example of what instructions should be given to the pilot if this condition is detected."

[1377] The above is an embodiment of the present invention, which can improve the safety of aircraft and effectively reduce the mental burden on pilots.

[1378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1379] Step 1:

[1380] The server obtains flight data in real time from the aircraft's sensors. Specifically, data such as the aircraft's altitude, speed, geographic coordinates, and engine status are obtained from the aircraft's central system using a dedicated protocol. The input data is raw data from the sensors, which is collected in real time and stored in a database. As an output, various flight data are stored in the database in an organized manner.

[1381] Step 2:

[1382] The server acquires video data from surveillance cameras within the airport and collects audio data from radio communications with air traffic controllers. The video data is received in streaming format, and the audio data is received in digital audio format. The video data is run through an image recognition algorithm to detect foreign objects on the runway, and the audio data is run through voice recognition technology to check for erroneous instructions. The output is compiled into a report containing the foreign object detection results and audio analysis results.

[1383] Step 3:

[1384] The server collects the pilot's voice and physiological information and provides it to the emotion engine. Voice is captured through a microphone, and physiological data is acquired through a heart rate sensor and skin potential sensor. The emotion engine analyzes this data and recognizes the pilot's emotional state. The input is the voice and physiological data collected from the pilot, and the output is the pilot's emotional state (e.g., high stress, relaxed).

[1385] Step 4:

[1386] The flight data, video data, audio data, and emotional data acquired by the server are input into the AI ​​model. The AI ​​model includes a neural network that integrates and analyzes this data. The input data is initial data from various sensors, and the output is a report containing the integrated analysis results (e.g., sudden weather changes and risk factors). Specifically, the model performs functions such as detecting anomalies in flight data, detecting foreign objects in video data, and detecting erroneous instructions in audio data.

[1387] Step 5:

[1388] Based on the analysis results, the server provides instructions to the pilot in text and voice format. Instructions are generated based on the results of the emotion engine, and the pilot is given appropriate operational guidance. Analysis results and emotion data are used as input data, and the output is voice instructions (e.g., "Please remain calm and maintain your current altitude") and text instructions displayed on the pilot's display.

[1389] Step 6:

[1390] The server analyzes flight data and emotional data in real time to detect emergencies. If an engine abnormality, a sudden change in weather, or a pilot's physiological abnormality is detected, an emergency alert is issued immediately. The input is real-time flight data and emotional data, and the output is a warning message about the emergency and detailed information.

[1391] Step 7:

[1392] When the server detects an emergency, it activates the autopilot system. The autopilot system then takes appropriate action based on pre-programmed emergency response scenarios. For example, if an engine malfunction is detected, an emergency landing route to the nearest airport is automatically set and the flight is carried out along that route. The input is the emergency detection information, and the output is the activation of the autopilot system and a log of its actions.

[1393] Step 8:

[1394] The server continuously accumulates collected flight data and uses it to improve the performance of the AI ​​model. The newly acquired data is used to retrain the model, enabling more accurate analysis. The input is continuously collected flight data, and the output is an updated AI model and its performance evaluation results. For example, it is possible to learn new patterns of abnormal situations and incorporate appropriate countermeasures into the system.

[1395] (Application example 2)

[1396] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1397] In modern manufacturing, improving the efficiency and safety of factory operations are important challenges. In particular, there is a need to monitor the work environment and worker status in real time and respond quickly and appropriately when an abnormality occurs. However, existing systems lack the technology to integrate and analyze multiple data sources and automatically generate work instructions, leaving room for improvement. Therefore, there is a need for a system that can efficiently integrate and analyze real-time data and provide work instructions, including emergency responses.

[1398] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring flight data, means for analyzing video and audio data, means for generating aircraft operation instructions based on the analysis results, means for detecting an emergency, means for activating an autopilot system when the emergency is detected, means for continuously training a machine learning model using the flight data and environmental data, means for acquiring sensor data within the factory, means for analyzing environmental data and physiological data, means for generating work instructions based on the analysis results, means for autonomously responding to emergencies based on the results of the analysis means, and means for updating the generating AI model and optimizing the instruction provision. This enables efficient work management and improved safety within the factory.

[1399] "Flight data" is data including aircraft altitude, speed, geographic coordinates, and engine status.

[1400] "Video and audio data" refers to data that includes video and audio from a surveillance camera.

[1401] "Aircraft operation instructions" are instructions regarding flight operations sent from the server to the pilot.

[1402] "Emergency" means an abnormal or dangerous condition in aircraft operations or factory operations.

[1403] An "autopilot system" is a system that controls the automatic operation of an aircraft.

[1404] A "machine learning model" is an artificial intelligence model that is trained to perform analysis and predictions using collected data.

[1405] "Sensor data" is data obtained from various sensors within the factory (temperature, humidity, machine operating status, etc.).

[1406] "Environmental data" refers to data including environmental conditions such as temperature, humidity, noise, and vibration within a factory.

[1407] "Physiological data" refers to data that indicates the physiological state of a worker, such as heart rate and skin potential.

[1408] "Work instructions" are specific instructions for actions provided to workers based on the analysis results.

[1409] A "generative AI model" is an artificial intelligence model that generates new instructions and countermeasures by analyzing data.

[1410] "Autonomous emergency response" means that the system automatically takes action to deal with an emergency.

[1411] As an embodiment of the present invention, the system is described in detail below. This system collects and analyzes flight data, video and audio data, sensor data, environmental data, physiological data, etc. in real time, and issues appropriate instructions and emergency responses based on the analysis results.

[1412] System Configuration

[1413] 1. Real-time data acquisition

[1414] The server receives real-time flight data from various sensors on the plane, including altitude, speed, geographic coordinates, and engine status, as well as sensor data from the factory, such as temperature, humidity, and machine operation status.

[1415] 2. Analysis of video and audio data

[1416] The server acquires surveillance camera footage and environmental audio, and uses video recognition technology to extract important visual information and audio recognition technology to detect signs of abnormalities.

[1417] 3. Physiological Data Collection and Analysis

[1418] The server collects physiological data such as the worker's heart rate and provides it to the emotion engine to analyze the worker's stress level. This analysis determines the worker's workload and takes the necessary measures.

[1419] 4. Data analysis using AI models

[1420] The server inputs the acquired flight data, video and audio data, sensor data, and physiological data into the AI ​​model, which then integrates and analyzes this data to identify risk factors both inside and outside the factory, contributing to improved safety and efficiency.

[1421] 5. Instruction Generation and Delivery

[1422] Based on the analysis, the server generates appropriate instructions for an aircraft pilot or a factory worker, delivered in text or audio format and including specific advice based on the worker's emotional state.

[1423] 6. Emergency Detection and Autonomous Response

[1424] The server uses the analysis data to detect abnormalities and emergencies in real time, and activates the autopilot system or the factory's emergency response system as necessary, allowing for a quick and safe response.

[1425] 7. Continuous learning

[1426] The server continuously updates the machine learning model using the collected data to improve the performance of the entire system, allowing it to quickly incorporate new abnormal patterns.

[1427] Hardware and software used

[1428] The hardware used is as follows:

[1429] Sensors: Various sensors that collect environmental data within the factory in real time

[1430] Camera: A surveillance camera for capturing video data

[1431] Physiological data monitor: A monitor that acquires information such as the worker's heart rate

[1432] The software used is as follows:

[1433] Pandas: Real-time data acquisition in dataframe format

[1434] OpenCV: Camera image analysis

[1435] Librosa: Analysis of audio data

[1436] Scikit-learn: Building and analyzing machine learning models

[1437] Specific examples

[1438] For example, if a factory machine detects abnormal vibrations, the system will immediately analyze the vibration data and provide appropriate instructions to workers. If a worker's heart rate is abnormally high, the system will instruct them to stop work and take a break to protect their health.

[1439] Prompt Sentence Examples

[1440] Here are some example prompts to input to the generative AI model:

[1441] "Sensor data, camera footage, and audio data from within the factory are collected in real time, and workers' physiological data is analyzed to detect abnormalities. All data is analyzed using AI models, risk assessment is performed, and appropriate instructions are provided to workers."

[1442] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1443] Step 1:

[1444] The server obtains flight data such as altitude, speed, geographic coordinates, and engine status from various sensors on the aircraft in real time. In addition, it also collects sensor data such as temperature, humidity, and machine operation status inside the factory. The input is sensor data, and the output is collected data in data frame format. The data is managed using the Pandas library.

[1445] Step 2:

[1446] The server acquires surveillance camera footage and factory audio data, and uses video and audio recognition technologies to detect abnormalities. The video data is analyzed using the OpenCV library, and the audio data is analyzed using the Librosa library. The input is the camera footage and audio data, and the output is a flag indicating whether or not an abnormality is present.

[1447] Step 3:

[1448] The server collects physiological data such as the worker's heart rate in real time and inputs it into the emotion engine. The emotion engine analyzes the biosignals and evaluates the worker's stress level. The input is physiological data and the output is stress level. Specifically, the emotional state is estimated from the heart rate and skin potential response.

[1449] Step 4:

[1450] The server inputs all collected flight data, video and audio data, sensor data, and physiological data into an AI model for analysis. The AI ​​model then integrates this data and extracts risk factors inside and outside the factory. The input is all collected data, and the output is the risk assessment results. The machine learning model is trained using Scikit-learn.

[1451] Step 5:

[1452] The server generates and notifies appropriate instructions to aircraft pilots and factory workers based on the analysis results. Instructions are provided in text and voice format. The input is the analysis result of the AI ​​model, and the output is the instruction content. Instructions are generated taking into account the results of the emotion engine.

[1453] Step 6:

[1454] The server detects abnormalities and emergencies in real time from the analysis data and activates the autopilot system or the factory's emergency response system as necessary. The input is real-time analysis data, and the output is the emergency state and the corresponding action. Specifically, when an abnormality is detected, the response system is immediately activated.

[1455] Step 7:

[1456] The server uses the collected data to continuously update the machine learning model, improving overall system performance. This allows new abnormal patterns to be quickly incorporated. The input is past collected data and new data, and the output is an updated AI model. New algorithms and parameter adjustments are also applied to the data as it is learned.

[1457] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1459] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1460] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1461] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1462] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1463] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1464] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1465] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1466] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1467] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1468] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1469] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1470] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1471] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1472] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1473] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1474] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1475] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1476] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1477] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1478] The following is further disclosed regarding the above embodiment.

[1479] (Claim 1)

[1480] a means for acquiring flight data;

[1481] means for analyzing the video and audio data;

[1482] means for generating aircraft operation instructions based on the analysis results;

[1483] a means for detecting an emergency;

[1484] means for activating an autopilot system upon detecting the emergency;

[1485] means for continuously training a machine learning model using the flight data;

[1486] A system including:

[1487] (Claim 2)

[1488] 10. The system of claim 1, wherein aircraft operation instructions based on the analysis results are provided to the pilot.

[1489] (Claim 3)

[1490] 10. The system of claim 1, wherein the flight data includes altitude, speed, geographic coordinates, and engine status.

[1491] "Example 1"

[1492] (Claim 1)

[1493] a means for acquiring flight data;

[1494] means for analyzing surveillance camera and wireless communication data;

[1495] means for generating aircraft operation instructions based on the analysis results;

[1496] a means for detecting an emergency;

[1497] means for activating an autopilot system upon detecting the emergency;

[1498] means for continuously training a machine learning model using the flight data and analysis data;

[1499] A system including:

[1500] (Claim 2)

[1501] 10. The system of claim 1, wherein aircraft operation instructions based on the analysis results are provided to the pilot.

[1502] (Claim 3)

[1503] 10. The system of claim 1, wherein the flight data includes altitude, speed, geographic coordinates, and engine status.

[1504] "Application Example 1"

[1505] (Claim 1)

[1506] a means for acquiring flight data;

[1507] means for analyzing the video and audio data;

[1508] means for generating aircraft operation instructions based on the analysis results;

[1509] a means for detecting an emergency;

[1510] means for activating an autopilot system upon detecting the emergency;

[1511] means for continuously training a machine learning model using the flight data;

[1512] A means of obtaining various data within the logistics center in real time,

[1513] a means for providing instructions to a worker and an automatic transport robot based on the analysis results;

[1514] A means for automatically executing emergency responses in the event of an abnormality;

[1515] A system including:

[1516] (Claim 2)

[1517] 10. The system of claim 1, wherein aircraft operation instructions based on the analysis results are provided to the pilot.

[1518] (Claim 3)

[1519] 10. The system of claim 1, wherein the flight data includes altitude, speed, geographic coordinates, and engine status.

[1520] "Example 2: Combining Emotion Engines"

[1521] (Claim 1)

[1522] a means for obtaining flight information;

[1523] means for analyzing the video and audio information;

[1524] means for generating aircraft operation instructions based on the analysis results;

[1525] a means for detecting an emergency;

[1526] means for activating an autopilot system upon detecting the emergency;

[1527] means for continuously training a machine learning model using the flight information;

[1528] means for analyzing the emotional state of the pilot from physiological and vocal information;

[1529] means for generating appropriate aircraft operation instructions using the emotional state analysis results;

[1530] A system including:

[1531] (Claim 2)

[1532] 10. The system of claim 1, wherein aircraft operation instructions based on the analysis results are provided to the pilot.

[1533] (Claim 3)

[1534] 10. The system of claim 1, wherein the flight information includes altitude, speed, geographic coordinates, and engine status.

[1535] "Application example 2 when combining emotion engines"

[1536] (Claim 1)

[1537] a means for acquiring flight data;

[1538] means for analyzing the video and audio data;

[1539] means for generating aircraft operation instructions based on the analysis results;

[1540] a means for detecting an emergency;

[1541] means for activating an autopilot system upon detecting the emergency;

[1542] means for continuously training a machine learning model using the flight data and environmental data;

[1543] A means of acquiring sensor data within the factory;

[1544] means for analyzing the environmental and physiological data;

[1545] A means for generating work instructions based on the analysis results;

[1546] means for autonomously taking emergency action based on the results of said analysis means;

[1547] a means for updating the generative AI model to optimize instruction delivery; and

[1548] A system including:

[1549] (Claim 2)

[1550] The system according to claim 1, wherein aircraft operation instructions and work instructions obtained based on the analysis results are provided to pilots and workers.

[1551] (Claim 3)

[1552] 10. The system of claim 1, wherein the flight data includes altitude, speed, geographic coordinates, and engine status. [Explanation of symbols]

[1553] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring flight data; means for analyzing the video and audio data; means for generating aircraft operation instructions based on the analysis results; a means for detecting an emergency; means for activating an autopilot system upon detecting the emergency; means for continuously training a machine learning model using the flight data; A system including:

2. The system of claim 1, further comprising: providing aircraft operation instructions to the pilot based on the analysis results.

3. 10. The system of claim 1, wherein the flight data includes altitude, speed, geographic coordinates, and engine status.

Images