System

Abstract

A system is provided.SOLUTION: A system comprising: receiving means for receiving wireless traffic in an aircraft in real time; voice recognition means for converting received voice data into text data; generating means for visualizing a situation of the aircraft based on the text data; displaying means for displaying the visualized situation data; and storing means for storing all voice and text data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Air traffic control is based on human judgment, with communication between the control tower and aircraft via voice radio, but there is always a risk of mishearing or misunderstanding. Such risks can lead to serious aviation accidents, and the risk is particularly high for small aircraft flown by a single pilot. A system is needed to reduce these risks as much as possible and improve safety. [Means for solving the problem]

[0005] The present invention provides a system that recognizes voice communications from radio communications and visualizes the contents of the communications to help pilots recognize misunderstandings. Specifically, the system includes a receiving means for receiving radio communications within an aircraft in real time, a voice recognition means for converting the received voice data into text data, a generating means for visualizing the aircraft's status based on the text data, a display means for displaying the visualized status data, and a storage means for storing all voice and text data. This system can prevent mishearing and misunderstandings during voice communications and reduce the risk of major accidents.

[0006] "Receiving means" refers to the equipment and functionality for receiving radio communications within the aircraft in real time.

[0007] "Speech recognition means" refers to devices and functions for converting received voice data into text data.

[0008] "Generation means" refers to a device and function for generating information that visualizes the status of an aircraft based on text data.

[0009] "Display means" refers to a display or display device for presenting visualized situational data to the pilot.

[0010] "Storage Means" means the equipment and facilities for storing all audio and text data for future analysis and review. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0033] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0034] The terminal then receives the voice data sent from the server and converts the voice data into text data using a voice recognition means, which uses a pre-trained voice recognition model.

[0035] The text data generated by the speech recognition means is processed by a generation means, which analyzes the text data and generates visualization data representing the aircraft's current operational status and instructions, including details such as flight number, current altitude, target altitude, and descent instructions.

[0036] The terminal then retrieves the visualization data generated by the generating means and displays it to the pilot using the displaying means, which may be a display or other suitable display device.

[0037] Additionally, the server stores all audio data and text data generated by speech recognition in local storage using a storage means, making the data available for future analysis and verification.

[0038] To give a specific example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." The terminal displays this visualization data on a display, allowing the pilot to visually confirm the content of the instruction, reducing mishearing and misrecognition.

[0039] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0040] The processing flow will be explained below.

[0041] Step 1:

[0042] The server captures voice data from the aircraft's radio communication system in real time, and after receiving the voice data, stores it in a temporary buffer, allowing the radio communication content to proceed to the next processing step without loss.

[0043] Step 2:

[0044] The terminal receives voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0045] Step 3:

[0046] After the text data is generated, the generating means of the terminal receives the text data and starts analyzing it, classifying the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generating appropriate visualization data.

[0047] Step 4:

[0048] The terminal receives the generated visualization data and transmits it to the display means, which displays the generated visualization data to the pilot using a display or other suitable display device, thereby allowing the pilot to visually confirm the current operational status and instructions.

[0049] Step 5:

[0050] The server will store all audio data and text data generated by speech recognition in local storage using a storage method, ensuring that the data is available for future analysis and verification.

[0051] In this way, a series of processing steps results in a system that accurately captures the contents of radio communications and provides them visually to the pilot, thereby reducing mishearing and misinterpretation of radio communications and improving aviation safety.

[0052] Example 1

[0053] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0054] Radio communication within an aircraft is important for pilots to receive instructions in real time, but voice communication is prone to misunderstandings and mishearing. Therefore, it is necessary to improve recognition accuracy by converting voice data into text data and visualizing it. However, current systems have difficulty achieving real-time and reliable voice recognition and data visualization.

[0055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0056] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the status of the aircraft based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, analysis means for extracting the operating status and instructions of the aircraft, and communication means for transmitting voice data from the server to the terminal. This enables real-time text conversion and visualization of voice data, reducing pilot judgment errors and ensuring safety.

[0057] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0058] "Speech recognition means" refers to a technique for converting received voice data into text data.

[0059] The "generation means" is a technology for visualizing the aircraft situation based on text data.

[0060] "Display means" means a device for displaying visualized situation data to the pilot.

[0061] "Storage Means" means any device or technology used to store all audio and text data.

[0062] The "analysis means" is a technology for extracting aircraft operating status and instruction content from text data.

[0063] "Communication means" refers to the technology for transmitting audio data from the server to the terminal.

[0064] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0065] The server captures voice data from the aircraft's radio communication system in real time. Specifically, the server acquires the voice data through a radio receiver installed on the aircraft and temporarily stores it in a buffer. For example, a high-performance radio receiver combined with dedicated data capture software can be used.

[0066] Next, the terminal receives the voice data sent from the server. At this time, the terminal converts the voice data into text data using a voice recognition means. As the voice recognition means, an existing voice recognition service such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text can be used.

[0067] The terminal receives the text data generated by speech recognition and analyzes it using an analysis means. This analysis uses a Python program or a natural language processing library (e.g., SpaCy, NLTK). The analysis means extracts the aircraft's operational status and instructions from the text data.

[0068] The terminal then generates visualization data based on the extracted information using a generation means. The visualization data includes information such as the flight number, current altitude, target altitude, and descent instructions. The generated visualization data is sent to an appropriate display device, such as a display or tablet, and displayed to the pilot. The display means can be a high-resolution display or a mobile tablet.

[0069] Furthermore, the server stores all voice data and text data generated by speech recognition in local storage (SSD or HDD) using a storage means, making the data available for future analysis and verification. For example, using a high-speed, reliable SSD allows for efficient data storage and retrieval.

[0070] As a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The device receives this voice data and converts it into text data "Flight 123, descend to 10,000 feet" using the Google Cloud Speech-to-Text API. The device then analyzes this text data using the SpaCy library and extracts information such as the flight number "Flight 123" and the intended descent altitude "10,000 feet." The device then generates visualization data based on the extracted information and displays it on the display. The pilot then checks the visualization data on the display and recognizes the descent instruction.

[0071] Example prompts to input to a generative AI model:

[0072] "Convert the audio data of the following radio communication into text data and generate a visualization. Audio data: "Flight 123, descend to 10,000 feet""

[0073] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0074] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0075] Step 1:

[0076] The server captures voice data in real time from the wireless communication system in the aircraft. Using a receiving means, it captures the voice signal transmitted from the wireless communication device and converts it into digital voice data. The input is the analog voice signal from the wireless communication, and the output is the digital voice data. This data is temporarily stored in a buffer.

[0077] Step 2:

[0078] The terminal receives the voice data sent from the server. The terminal receives the voice data from the server using a communication means and temporarily stores it in local storage. The input is the digital voice data sent from the server, and the output is the voice data stored in the local storage.

[0079] Step 3:

[0080] The device converts the received voice data into text data using a voice recognition means. Using a voice recognition means (e.g., Google Cloud Speech-to-Text API), the voice data is analyzed and converted into text format. The input is the voice data stored in local storage, and the output is the text data generated by voice recognition.

[0081] Step 4:

[0082] The terminal analyzes the generated text data using an analysis means. A natural language processing library (e.g., SpaCy, NLTK) is used to extract the aircraft's operational status and instructions from the text data. The input is the text data generated by the speech recognition means, and the output is the extracted operational status and instructions. Specifically, the terminal extracts keywords such as flight number and altitude from the text and performs analysis based on these.

[0083] Step 5:

[0084] The terminal generates visualization data using a generation means based on the extracted information. The generated visualization data includes information such as the flight number, current altitude, destination altitude, and descent instructions. The input is the flight status and instructions extracted by the analysis means, and the output is data that can be displayed visually. Specifically, the terminal generates a graphical interface based on the extracted information and structures it in an appropriate format.

[0085] Step 6:

[0086] The terminal displays the generated visualization data to the pilot using a display means. The display means may be a high-resolution display or a mobile tablet. The input is the visualization data from the generation means, and the output is the data displayed on the display device. Specific operations include controlling the layout and display of the data on the display.

[0087] Step 7:

[0088] The server uses a storage means to store all voice data and text data generated by voice recognition in local storage. The input is the captured voice data and text data, and the output is the data stored in local storage. Specifically, the data is stored in a high-speed, reliable storage device (SSD or HDD) and made available for later analysis and verification.

[0089] (Application example 1)

[0090] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In current logistics centers, when workers receive work instructions via voice, efficiency can decrease due to mishearing or lack of recognition. Manual confirmation of work instructions also poses a problem, resulting in work delays. In particular, when workers work for long periods of time while connected to a wireless network, receiving continuous instructions increases the risk of overwork. We propose a new system to solve these issues and provide an efficient and accurate method of transmitting work instructions.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0093] In this invention, the server includes a receiving means for receiving wireless communications in real time, a voice recognition means for converting received voice data into text data, a generating means for visualizing a situation based on the text data, a display means for displaying the visualized situation data, and a storage means for saving all voice and text data, thereby enabling workers to receive accurate work instructions in real time through smart glasses or a head-mounted display, and to work efficiently.

[0094] "Wireless communication" is a technology for transmitting and receiving information such as voice data and text data using wireless signals.

[0095] The "receiving means" is a device or system for receiving voice data and other data transmitted by wireless communication in real time.

[0096] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0097] The "generation means" is a technology or device for analyzing a situation based on the converted text data and generating visualization data.

[0098] "Display means" refers to a technology or device for displaying the generated visualization data to the worker. This includes smart glasses and head-mounted displays.

[0099] "Storage means" refers to technology or devices for storing received voice data and generated text data.

[0100] A "generative AI model" is an artificial intelligence model that analyzes situations based on text data and visualizes work instructions.

[0101] The present invention relates to a system that allows workers at a logistics center to visually obtain work instructions in real time using smart glasses or a head-mounted display.

[0102] 1. Capture audio data

[0103] The server receives voice data in real time using a wireless communication system, specifically, captures voice data including work instructions through a wireless communication device, and temporarily stores the captured voice data in a buffer.

[0104] 2. Speech Recognition and Text Conversion

[0105] The device converts the voice data sent from the server into text data in real time using a speech recognition method. The speech recognition method uses the "speech_recognition" library. The voice data is converted into text through the Google Speech Recognition API.

[0106] 3. Analysis and visualization of text data

[0107] The generation means analyzes the converted text data and generates visualization data containing specific work instructions. Here, a generative AI model is used to convert the work instructions based on the text data into a form that is visually easy to understand. This generative AI model uses, for example, a natural language processing model for text analysis.

[0108] 4. Displaying Visualized Data

[0109] The device then displays the generated visualization data on smart glasses or a head-mounted display, allowing workers to view the visualized work instructions in real time. Examples of display methods include Google Glass or similar smart glasses.

[0110] 5. Data storage

[0111] The server uses a storage means to store all received audio data and generated text data in local storage, ensuring data availability for future analysis and verification.

[0112] Specific examples

[0113] As a concrete example, let's consider the case where an instruction is given via wireless communication within a logistics center: "Pick up Box 42 from Zone 5 and move it to Zone 10." The server receives this voice data in real time and temporarily stores it. The device uses the Google Speech Recognition API to convert this voice data into text data: "Pick up Box 42 from Zone 5 and move it to Zone 10."

[0114] Next, the generation means analyzes this text data and generates visualization data as specific work instructions, such as "Pick up Box 42 from Zone 5 and move it to Zone 10." The worker's smart glasses display this visualization data, allowing the worker to check it in real time.

[0115] Examples of prompt statements

[0116] Listening for instructions...

[0117] Instructions: Pick up Box 42 from Zone 5 and move it to Zone 10

[0118] Recognized text: Pick up Box 42 from Zone 5 and move to Zone 10

[0119] Displaying instructions on smart glasses:

[0120] {

[0121] "timestamp": "2023-10-10 14:25:30",

[0122] "instruction": "Pick up Box 42 from Zone 5 and move it to Zone 10"

[0123] }

[0124] In this way, the system of the present invention can improve work efficiency in a logistics center and increase the accuracy of the transmission of work instructions.

[0125] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0126] Step 1:

[0127] The server receives voice data in real time through a wireless communication system. Specifically, it uses a wireless communication device to capture voice data such as work instructions. This voice data is temporarily stored in a buffer. The input is a wireless communication signal, and the output is the captured voice data.

[0128] Step 2:

[0129] The device receives the voice data sent from the server and converts it into text data using a speech recognition method. This speech recognition method uses the "speech_recognition" library and converts the voice data into text data via the Google Speech Recognition API. The input is voice data and the output is text data.

[0130] Step 3:

[0131] The terminal passes the converted text data to a generation means, which analyzes the text data and generates visualization data. Specifically, the generation means uses a natural language processing model to analyze the text data. The generated visualization data includes specific work instruction content. The input is text data, and the output is visualization data.

[0132] Step 4:

[0133] The terminal displays the generated visualization data on smart glasses or a head-mounted display. Google Glass or similar smart glasses are used as the display means. The input is the visualization data, and the output is the displayed work instructions. The user can confirm and carry out the work instructions through this display.

[0134] Step 5:

[0135] The server stores the received voice data and generated text data in local storage using a storage means, ensuring data for future analysis and verification. The input is the voice data and text data, and the output is the stored data.

[0136] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0137] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and further recognizes the user's emotions and displays additional information based on these. The specific configuration and processing of the system are described below.

[0138] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0139] Next, the terminal receives the voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0140] The text data generated by the speech recognition means is processed by the generation means, which classifies the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generates appropriate visualization data.

[0141] Furthermore, the emotion engine analyzes the user's emotions based on the voice data. The results of this emotion analysis are also reflected in the visualization data, enriching the content presented to the pilot. For example, it is possible to respond by displaying an alert if there are large fluctuations in emotions.

[0142] The emotion analysis data generated by the emotion engine is also stored as text data in the storage means, ensuring data availability for future analysis and verification.

[0143] To give a concrete example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes the user's emotions, and if there are signs of tension or fatigue, for example, that information is also displayed. The terminal displays this visualization data on a display, allowing the pilot to visually confirm the instructions and their own emotional state.

[0144] In this way, the system of the present invention not only improves the recognition accuracy of radio communications on board an aircraft, but also provides information that takes into account the user's emotional state, thereby further improving safety.

[0145] The processing flow will be explained below.

[0146] Step 1:

[0147] The server captures voice data from the aircraft's radio communication system in real time and temporarily stores the captured voice data in a buffer, ensuring that the content of the radio communication is accurately used for subsequent processing.

[0148] Step 2:

[0149] The terminal receives the voice data sent from the server. The received voice data is input into the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0150] Step 3:

[0151] The generating means analyzes the text data obtained by the speech recognition means, extracts aircraft operation information, such as flight number, altitude, speed, etc., from the text data, and generates visualization data.

[0152] Step 4:

[0153] The device's emotion engine analyzes the user's emotions in real time from voice data. Emotions are evaluated using multiple indicators, including tension, fatigue, and stress. Based on the analysis results, an alert is generated if there are large fluctuations in emotions.

[0154] Step 5:

[0155] The terminal integrates the visualization data obtained by the generating means with the emotion data analyzed by the emotion engine, and transmits the integrated information to the display means for visually displaying to the pilot.

[0156] Step 6:

[0157] The display means of the terminal uses a device such as a display to display the integrated visualization data and emotion data to the pilot, thereby enabling the pilot to quickly and accurately grasp the instructions and his / her own emotional state.

[0158] Step 7:

[0159] The server stores all audio and text data, as well as sentiment analysis data, in local storage using a storage means, ensuring data is available for future analysis and validation.

[0160] To give a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal uses a voice recognition means to convert the voice data into text data such as "Flight 123, descend to 10,000 feet." The generation means analyzes the text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes signs of tension or fatigue in the pilot's voice and generates this as emotion data. The terminal's display means displays this visualization data and emotion data on a display, allowing the pilot to visually confirm the instruction and his or her own emotional state. All of this information is stored by the server and can be used for later analysis and verification.

[0161] Example 2

[0162] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] Radio communications on aircraft must be understood accurately and quickly, especially in emergencies. However, simply converting speech to text can lead to information omissions and misunderstandings. Furthermore, providing information without considering the psychological state of the pilot and crew can lead to further risks. Therefore, there is a need for a comprehensive information system that accurately recognizes radio communications and takes into account the user's emotional state.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0165] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting the received voice data into text data, generation means for analyzing the converted text data and generating visualized data of the aircraft situation, emotion analysis means for analyzing the user's emotions from the voice data, display means for displaying the visualized situation data and the emotion analysis results, and storage means for storing all the voice data, text data, and the emotion analysis results, thereby enabling accurate recognition of wireless communications and provision of information taking into account the user's emotions.

[0166] "Receiving means" means a device or function for receiving radio communications within an aircraft in real time.

[0167] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0168] The "generation means" is a technology or device for analyzing the converted text data and generating visualization data of the aircraft situation.

[0169] "Emotion analysis means" refers to a technique or device for analyzing a user's emotions from voice data.

[0170] The "display means" is a device or function for displaying visualized situation data and emotion analysis results.

[0171] "Storage means" refers to the technology or device for storing all audio data, text data, and sentiment analysis results.

[0172] A "wireless communication system" is a system for wireless communication within an aircraft.

[0173] The system of the present invention receives wireless communications within an aircraft in real time, converts the voice data into text data for visualization, and further recognizes the user's emotions and displays additional information based on the emotions. This system includes a receiving means, a voice recognition means, a generating means, an emotion analysis means, a display means, and a storage means.

[0174] First, the server captures voice data in real time from the aircraft's radio communication system. As a receiving means, a device with a wireless module is used, which ensures reliable reception of radio communications within the aircraft. The received voice data is immediately stored in a buffer.

[0175] Next, the device receives the voice data sent from the server. Using wireless communication technologies such as Wi-Fi or Bluetooth, the device captures this voice data locally. This data is converted into text data by speech recognition using the Google Speech-to-Text API.

[0176] The converted text data is then parsed by a generator and categorized into flight information such as flight number, altitude, speed, etc. This analysis process uses Python scripts and other data analysis tools, and this information is then generated into visualizations that allow users to understand the aircraft's flight status at a glance.

[0177] Additionally, Microsoft Azure's Text Analytics API is used for sentiment analysis. Pilot emotions are analyzed from voice and text data to detect signs of tension or fatigue. The results of this sentiment analysis are also reflected in the visualized data, providing pilots with richer information.

[0178] All generated data, including audio data, text data, and sentiment analysis results, are stored in a database by a storage means, and this storage process is designed to be available for future analysis and validation.

[0179] The visualized data and the results of the emotion analysis are displayed on the device's display. For example, if the command "Flight 123, descend to 10,000 feet" is received, this is converted into text data in real time and the visualized data "Flight 123 has been instructed to descend to an altitude of 10,000 feet" is displayed. At the same time, if any signs of tension are shown, that information is also displayed.

[0180] An example prompt is:

[0181] "If a radio command such as 'Flight 123, descend to 10,000 feet' is received, explain how you would convert the audio data into text and visualize it. Also, explain how you would incorporate sentiment analysis results into the visualization."

[0182] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0183] Step 1:

[0184] The server receives real-time voice data from the aircraft's radio communication system. Specifically, it uses a radio module to capture radio communication within the aircraft. The input is the voice signal captured from the aircraft's radio frequency, and the output is the digitized voice data.

[0185] Step 2:

[0186] The server temporarily stores the received audio data in a buffer, which prevents data loss or delay. The input is digitized audio data, and the output is audio data stored in the buffer in memory.

[0187] Step 3:

[0188] The device receives audio data from the server. Specifically, the audio data is transferred via Wi-Fi or Bluetooth. The input is the audio data transmitted wirelessly from the server, and the output is the audio data stored in the device's temporary storage.

[0189] Step 4:

[0190] The device converts the received voice data into text data using a voice recognition method. Using the Google Speech-to-Text API, the audio file is sent to the API to obtain text data. The input is the voice data stored on the device, and the output is the text data converted from the voice data.

[0191] Step 5:

[0192] The terminal analyzes the text data using a generation means and generates visualization data of aircraft operation status. Using a Python script, it extracts information such as flight number, altitude, and speed and generates this into graphs and icons. The input is the text data generated by the speech recognition means, and the output is the visualization data.

[0193] Step 6:

[0194] The device uses emotion analysis to analyze the user's emotions. It uses Microsoft Azure's Text Analytics API to send text data to the API and obtains the emotion analysis results. The input is voice or text data, and the output is analysis data that indicates the user's emotional state.

[0195] Step 7:

[0196] The device reflects the results of the emotion analysis in the visualized data. Based on the results of the emotion analysis, alerts and attention-grabbing elements are added to the visualized data. The input is the emotion data generated by the emotion analysis means, and the output is the highlighted visualized data.

[0197] Step 8:

[0198] The device stores all generated data in a storage means. Specifically, it stores voice data, text data, and emotion analysis results in a database. The input is the visualization data and emotion analysis data, and the output is the data stored in the database.

[0199] Step 9:

[0200] The terminal displays the visualized data and emotion analysis results on a display. The pilot can visually confirm the instructions and his / her own emotional state. The input is the data obtained from the generation means and emotion analysis means, and the output is the visualized information and emotion information displayed on the display.

[0201] (Application example 2)

[0202] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0203] Conventional systems can receive aircraft radio communications in real time and visualize their content, but they are unable to provide information that takes into account the user's emotional state. This has led to the problem that pilots and security staff are unable to respond appropriately when faced with an emergency or excessive stress. The present invention aims to solve this problem by providing a system that provides advanced information that takes into account the user's emotional state.

[0204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0205] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the aircraft's status based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, emotion analysis means for analyzing the user's emotions, and alert display means for displaying an alert based on the emotion analysis result, thereby making it possible to visualize the contents of wireless communications and display an alert according to the user's emotional state.

[0206] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0207] "Speech recognition means" refers to software or hardware for converting received voice data into text data.

[0208] A "generator" is a process or device for visualizing aircraft situations based on text data.

[0209] "Display means" refers to a display or other display device for showing visualized situation data to a user.

[0210] "Storage Means" means a data storage device for recording and archiving all audio and text data.

[0211] "Emotion analysis means" refers to software or artificial intelligence models for analyzing emotions from a user's voice or text.

[0212] The "alert display means" is a display device or system for notifying the user of necessary alerts based on the emotion analysis results.

[0213] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and also recognizes the user's emotions and displays additional information based on those emotions. The specific configuration and processing of this system will be described below.

[0214] First, the server captures voice data in real time from the aircraft's radio communication system. The captured voice data is temporarily stored in a buffer. The terminal then receives the voice data sent from the server, inputs this voice data into a voice recognition means, and converts it into text data. At this stage, the "speech_recognition" library is used to convert the voice data into text data, and then emotion analysis is performed using the "transformers" library and Hugging Face's "pipeline."

[0215] The generator analyzes the converted text data and appropriately visualizes the aircraft status data, specifically classifying the instructions and information converted from the voice into flight information such as flight number, altitude, speed, etc., using appropriately designed templates and dashboards.

[0216] Next, the sentiment analysis means analyzes the text and voice data to determine the user's emotional state. This process uses the Sentiment Analysis Pipeline. The results of the sentiment analysis evaluate the user's level of tension, fatigue, etc., and based on this, the system decides whether to display an alert.

[0217] The alert display means provides visual alerts to the user as needed based on the emotion analysis results. If abnormal emotion fluctuations or signs of danger are detected, an alert is displayed immediately. This alert display is performed using devices such as smart glasses or a head-mounted display.

[0218] For example, if a security staff member radios in to say, "An important person has entered the building," the system can convert this speech into text in real time and perform sentiment analysis. If the process determines that the staff member is overstressed, an appropriate alert will be displayed on the smart glasses.

[0219] Prompt Sentence Examples

[0220] Convert the speech "An important person has entered the building" into text, run it through a sentiment analysis model, and display the results.

[0221] This allows the system of the present invention to improve the safety and efficiency of aircraft operations and security operations by combining real-time speech-to-text conversion with emotional state recognition and providing information based on these.

[0222] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0223] Step 1:

[0224] The server receives voice data in real time from the aircraft's radio communication system. The received voice data is temporarily stored in a buffer. At this stage, the input is the voice data from the radio communication system, and the output is the voice data stored in the buffer.

[0225] Step 2:

[0226] The device receives the voice data sent from the server. It converts this voice data into text data using a speech recognition tool. This process uses the "speech_recognition" library. The input is the voice data in the buffer, and the output is the converted text data.

[0227] Step 3:

[0228] The generation means receives the text data obtained from the speech recognition means and analyzes the text data. During the analysis process, the text data is classified into aircraft operation information such as flight number, altitude, and speed, and appropriate visualization data is generated. The input is text data, and the output is visualization data. Templates and dashboards are used for this visualization data.

[0229] Step 4:

[0230] The emotion analysis method uses text and audio data to analyze the user's emotional state. The "transformers" library and Hugging Face's "pipeline" are used to perform emotion evaluation based on text and audio data. The input is text and audio data, and the output is the emotion analysis result.

[0231] Step 5:

[0232] Based on the emotion analysis results, the terminal displays an alert as necessary. The alert display means displays visual and auditory alerts to the user on an appropriate device (such as smart glasses or a head-mounted display) based on the emotion analysis results. The input is the emotion analysis results, and the output is the visual and auditory alerts.

[0233] Step 6:

[0234] The terminal displays the generated visualization data and emotion analysis results on a display, providing them to pilots and security staff. This allows users to grasp the aircraft's operational status and their own emotional state in real time. The input is the visualization data and emotion analysis results, and the output is the information displayed on the display.

[0235] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0236] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0237] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0238] [Second embodiment]

[0239] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0240] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0241] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0242] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0243] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0244] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0245] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0246] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0247] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0248] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0249] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0250] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0251] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0252] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0253] The terminal then receives the voice data sent from the server and converts the voice data into text data using a voice recognition means, which uses a pre-trained voice recognition model.

[0254] The text data generated by the speech recognition means is processed by a generation means, which analyzes the text data and generates visualization data representing the aircraft's current operational status and instructions, including details such as flight number, current altitude, target altitude, and descent instructions.

[0255] The terminal then retrieves the visualization data generated by the generating means and displays it to the pilot using the displaying means, which may be a display or other suitable display device.

[0256] Additionally, the server stores all audio data and text data generated by speech recognition in local storage using a storage means, making the data available for future analysis and verification.

[0257] To give a specific example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." The terminal displays this visualization data on a display, allowing the pilot to visually confirm the content of the instruction, reducing mishearing and misrecognition.

[0258] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The server captures voice data from the aircraft's radio communication system in real time, and after receiving the voice data, stores it in a temporary buffer, allowing the radio communication content to proceed to the next processing step without loss.

[0262] Step 2:

[0263] The terminal receives voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0264] Step 3:

[0265] After the text data is generated, the generating means of the terminal receives the text data and starts analyzing it, classifying the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generating appropriate visualization data.

[0266] Step 4:

[0267] The terminal receives the generated visualization data and transmits it to the display means, which displays the generated visualization data to the pilot using a display or other suitable display device, thereby allowing the pilot to visually confirm the current operational status and instructions.

[0268] Step 5:

[0269] The server will store all audio data and text data generated by speech recognition in local storage using a storage method, ensuring that the data is available for future analysis and verification.

[0270] In this way, a series of processing steps results in a system that accurately captures the contents of radio communications and provides them visually to the pilot, thereby reducing mishearing and misinterpretation of radio communications and improving aviation safety.

[0271] Example 1

[0272] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0273] Radio communication within an aircraft is important for pilots to receive instructions in real time, but voice communication is prone to misunderstandings and mishearing. Therefore, it is necessary to improve recognition accuracy by converting voice data into text data and visualizing it. However, current systems have difficulty achieving real-time and reliable voice recognition and data visualization.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0275] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the status of the aircraft based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, analysis means for extracting the operating status and instructions of the aircraft, and communication means for transmitting voice data from the server to the terminal. This enables real-time text conversion and visualization of voice data, reducing pilot judgment errors and ensuring safety.

[0276] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0277] "Speech recognition means" refers to a technique for converting received voice data into text data.

[0278] The "generation means" is a technology for visualizing the aircraft situation based on text data.

[0279] "Display means" means a device for displaying visualized situation data to the pilot.

[0280] "Storage Means" means any device or technology used to store all audio and text data.

[0281] The "analysis means" is a technology for extracting aircraft operating status and instruction content from text data.

[0282] "Communication means" refers to the technology for transmitting audio data from the server to the terminal.

[0283] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0284] The server captures voice data from the aircraft's radio communication system in real time. Specifically, the server acquires the voice data through a radio receiver installed on the aircraft and temporarily stores it in a buffer. For example, a high-performance radio receiver combined with dedicated data capture software can be used.

[0285] Next, the terminal receives the voice data sent from the server. At this time, the terminal converts the voice data into text data using a voice recognition means. As the voice recognition means, an existing voice recognition service such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text can be used.

[0286] The terminal receives the text data generated by speech recognition and analyzes it using an analysis means. This analysis uses a Python program or a natural language processing library (e.g., SpaCy, NLTK). The analysis means extracts the aircraft's operational status and instructions from the text data.

[0287] The terminal then generates visualization data based on the extracted information using a generation means. The visualization data includes information such as the flight number, current altitude, target altitude, and descent instructions. The generated visualization data is sent to an appropriate display device, such as a display or tablet, and displayed to the pilot. The display means can be a high-resolution display or a mobile tablet.

[0288] Furthermore, the server stores all voice data and text data generated by speech recognition in local storage (SSD or HDD) using a storage means, making the data available for future analysis and verification. For example, using a high-speed, reliable SSD allows for efficient data storage and retrieval.

[0289] As a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The device receives this voice data and converts it into text data "Flight 123, descend to 10,000 feet" using the Google Cloud Speech-to-Text API. The device then analyzes this text data using the SpaCy library and extracts information such as the flight number "Flight 123" and the intended descent altitude "10,000 feet." The device then generates visualization data based on the extracted information and displays it on the display. The pilot then checks the visualization data on the display and recognizes the descent instruction.

[0290] Example prompts to input to a generative AI model:

[0291] "Convert the audio data of the following radio communication into text data and generate a visualization. Audio data: "Flight 123, descend to 10,000 feet""

[0292] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0293] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0294] Step 1:

[0295] The server captures voice data in real time from the wireless communication system in the aircraft. Using a receiving means, it captures the voice signal transmitted from the wireless communication device and converts it into digital voice data. The input is the analog voice signal from the wireless communication, and the output is the digital voice data. This data is temporarily stored in a buffer.

[0296] Step 2:

[0297] The terminal receives the voice data sent from the server. The terminal receives the voice data from the server using a communication means and temporarily stores it in local storage. The input is the digital voice data sent from the server, and the output is the voice data stored in the local storage.

[0298] Step 3:

[0299] The device converts the received voice data into text data using a voice recognition means. Using a voice recognition means (e.g., Google Cloud Speech-to-Text API), the voice data is analyzed and converted into text format. The input is the voice data stored in local storage, and the output is the text data generated by voice recognition.

[0300] Step 4:

[0301] The terminal analyzes the generated text data using an analysis means. A natural language processing library (e.g., SpaCy, NLTK) is used to extract the aircraft's operational status and instructions from the text data. The input is the text data generated by the speech recognition means, and the output is the extracted operational status and instructions. Specifically, the terminal extracts keywords such as flight number and altitude from the text and performs analysis based on these.

[0302] Step 5:

[0303] The terminal generates visualization data using a generation means based on the extracted information. The generated visualization data includes information such as the flight number, current altitude, destination altitude, and descent instructions. The input is the flight status and instructions extracted by the analysis means, and the output is data that can be displayed visually. Specifically, the terminal generates a graphical interface based on the extracted information and structures it in an appropriate format.

[0304] Step 6:

[0305] The terminal displays the generated visualization data to the pilot using a display means. The display means may be a high-resolution display or a mobile tablet. The input is the visualization data from the generation means, and the output is the data displayed on the display device. Specific operations include controlling the layout and display of the data on the display.

[0306] Step 7:

[0307] The server uses a storage means to store all voice data and text data generated by voice recognition in local storage. The input is the captured voice data and text data, and the output is the data stored in local storage. Specifically, the data is stored in a high-speed, reliable storage device (SSD or HDD) and made available for later analysis and verification.

[0308] (Application example 1)

[0309] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0310] In current logistics centers, when workers receive work instructions via voice, efficiency can decrease due to mishearing or lack of recognition. Manual confirmation of work instructions also poses a problem, resulting in work delays. In particular, when workers work for long periods of time while connected to a wireless network, receiving continuous instructions increases the risk of overwork. We propose a new system to solve these issues and provide an efficient and accurate method of transmitting work instructions.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0312] In this invention, the server includes a receiving means for receiving wireless communications in real time, a voice recognition means for converting received voice data into text data, a generating means for visualizing a situation based on the text data, a display means for displaying the visualized situation data, and a storage means for saving all voice and text data, thereby enabling workers to receive accurate work instructions in real time through smart glasses or a head-mounted display, and to work efficiently.

[0313] "Wireless communication" is a technology for transmitting and receiving information such as voice data and text data using wireless signals.

[0314] The "receiving means" is a device or system for receiving voice data and other data transmitted by wireless communication in real time.

[0315] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0316] The "generation means" is a technology or device for analyzing a situation based on the converted text data and generating visualization data.

[0317] "Display means" refers to a technology or device for displaying the generated visualization data to the worker. This includes smart glasses and head-mounted displays.

[0318] "Storage means" refers to technology or devices for storing received voice data and generated text data.

[0319] A "generative AI model" is an artificial intelligence model that analyzes situations based on text data and visualizes work instructions.

[0320] The present invention relates to a system that allows workers at a logistics center to visually obtain work instructions in real time using smart glasses or a head-mounted display.

[0321] 1. Capture audio data

[0322] The server receives voice data in real time using a wireless communication system, specifically, captures voice data including work instructions through a wireless communication device, and temporarily stores the captured voice data in a buffer.

[0323] 2. Speech Recognition and Text Conversion

[0324] The device converts the voice data sent from the server into text data in real time using a speech recognition method. The speech recognition method uses the "speech_recognition" library. The voice data is converted into text through the Google Speech Recognition API.

[0325] 3. Analysis and visualization of text data

[0326] The generation means analyzes the converted text data and generates visualization data containing specific work instructions. Here, a generative AI model is used to convert the work instructions based on the text data into a form that is visually easy to understand. This generative AI model uses, for example, a natural language processing model for text analysis.

[0327] 4. Displaying Visualized Data

[0328] The device then displays the generated visualization data on smart glasses or a head-mounted display, allowing workers to view the visualized work instructions in real time. Examples of display methods include Google Glass or similar smart glasses.

[0329] 5. Data storage

[0330] The server uses a storage means to store all received audio data and generated text data in local storage, ensuring data availability for future analysis and verification.

[0331] Specific examples

[0332] As a concrete example, let's consider the case where an instruction is given via wireless communication within a logistics center: "Pick up Box 42 from Zone 5 and move it to Zone 10." The server receives this voice data in real time and temporarily stores it. The device uses the Google Speech Recognition API to convert this voice data into text data: "Pick up Box 42 from Zone 5 and move it to Zone 10."

[0333] Next, the generation means analyzes this text data and generates visualization data as specific work instructions, such as "Pick up Box 42 from Zone 5 and move it to Zone 10." The worker's smart glasses display this visualization data, allowing the worker to check it in real time.

[0334] Examples of prompt statements

[0335] Listening for instructions...

[0336] Instructions: Pick up Box 42 from Zone 5 and move it to Zone 10

[0337] Recognized text: Pick up Box 42 from Zone 5 and move to Zone 10

[0338] Displaying instructions on smart glasses:

[0339] {

[0340] "timestamp": "2023-10-10 14:25:30",

[0341] "instruction": "Pick up Box 42 from Zone 5 and move it to Zone 10"

[0342] }

[0343] In this way, the system of the present invention can improve work efficiency in a logistics center and increase the accuracy of the transmission of work instructions.

[0344] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0345] Step 1:

[0346] The server receives voice data in real time through a wireless communication system. Specifically, it uses a wireless communication device to capture voice data such as work instructions. This voice data is temporarily stored in a buffer. The input is a wireless communication signal, and the output is the captured voice data.

[0347] Step 2:

[0348] The device receives the voice data sent from the server and converts it into text data using a speech recognition method. This speech recognition method uses the "speech_recognition" library and converts the voice data into text data via the Google Speech Recognition API. The input is voice data and the output is text data.

[0349] Step 3:

[0350] The terminal passes the converted text data to a generation means, which analyzes the text data and generates visualization data. Specifically, the generation means uses a natural language processing model to analyze the text data. The generated visualization data includes specific work instruction content. The input is text data, and the output is visualization data.

[0351] Step 4:

[0352] The terminal displays the generated visualization data on smart glasses or a head-mounted display. Google Glass or similar smart glasses are used as the display means. The input is the visualization data, and the output is the displayed work instructions. The user can confirm and carry out the work instructions through this display.

[0353] Step 5:

[0354] The server stores the received voice data and generated text data in local storage using a storage means, ensuring data for future analysis and verification. The input is the voice data and text data, and the output is the stored data.

[0355] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0356] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and further recognizes the user's emotions and displays additional information based on these. The specific configuration and processing of the system are described below.

[0357] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0358] Next, the terminal receives the voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0359] The text data generated by the speech recognition means is processed by the generation means, which classifies the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generates appropriate visualization data.

[0360] Furthermore, the emotion engine analyzes the user's emotions based on the voice data. The results of this emotion analysis are also reflected in the visualization data, enriching the content presented to the pilot. For example, it is possible to respond by displaying an alert if there are large fluctuations in emotions.

[0361] The emotion analysis data generated by the emotion engine is also stored as text data in the storage means, ensuring data availability for future analysis and verification.

[0362] To give a concrete example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes the user's emotions, and if there are signs of tension or fatigue, for example, that information is also displayed. The terminal displays this visualization data on a display, allowing the pilot to visually confirm the instructions and their own emotional state.

[0363] In this way, the system of the present invention not only improves the recognition accuracy of radio communications on board an aircraft, but also provides information that takes into account the user's emotional state, thereby further improving safety.

[0364] The processing flow will be explained below.

[0365] Step 1:

[0366] The server captures voice data from the aircraft's radio communication system in real time and temporarily stores the captured voice data in a buffer, ensuring that the content of the radio communication is accurately used for subsequent processing.

[0367] Step 2:

[0368] The terminal receives the voice data sent from the server. The received voice data is input into the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0369] Step 3:

[0370] The generating means analyzes the text data obtained by the speech recognition means, extracts aircraft operation information, such as flight number, altitude, speed, etc., from the text data, and generates visualization data.

[0371] Step 4:

[0372] The device's emotion engine analyzes the user's emotions in real time from voice data. Emotions are evaluated using multiple indicators, including tension, fatigue, and stress. Based on the analysis results, an alert is generated if there are large fluctuations in emotions.

[0373] Step 5:

[0374] The terminal integrates the visualization data obtained by the generating means with the emotion data analyzed by the emotion engine, and transmits the integrated information to the display means for visually displaying to the pilot.

[0375] Step 6:

[0376] The display means of the terminal uses a device such as a display to display the integrated visualization data and emotion data to the pilot, thereby enabling the pilot to quickly and accurately grasp the instructions and his / her own emotional state.

[0377] Step 7:

[0378] The server stores all audio and text data, as well as sentiment analysis data, in local storage using a storage means, ensuring data is available for future analysis and validation.

[0379] To give a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal uses a voice recognition means to convert the voice data into text data such as "Flight 123, descend to 10,000 feet." The generation means analyzes the text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes signs of tension or fatigue in the pilot's voice and generates this as emotion data. The terminal's display means displays this visualization data and emotion data on a display, allowing the pilot to visually confirm the instruction and his or her own emotional state. All of this information is stored by the server and can be used for later analysis and verification.

[0380] Example 2

[0381] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0382] Radio communications on aircraft must be understood accurately and quickly, especially in emergencies. However, simply converting speech to text can lead to information omissions and misunderstandings. Furthermore, providing information without considering the psychological state of the pilot and crew can lead to further risks. Therefore, there is a need for a comprehensive information system that accurately recognizes radio communications and takes into account the user's emotional state.

[0383] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0384] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting the received voice data into text data, generation means for analyzing the converted text data and generating visualized data of the aircraft situation, emotion analysis means for analyzing the user's emotions from the voice data, display means for displaying the visualized situation data and the emotion analysis results, and storage means for storing all the voice data, text data, and the emotion analysis results, thereby enabling accurate recognition of wireless communications and provision of information taking into account the user's emotions.

[0385] "Receiving means" means a device or function for receiving radio communications within an aircraft in real time.

[0386] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0387] The "generation means" is a technology or device for analyzing the converted text data and generating visualization data of the aircraft situation.

[0388] "Emotion analysis means" refers to a technique or device for analyzing a user's emotions from voice data.

[0389] The "display means" is a device or function for displaying visualized situation data and emotion analysis results.

[0390] "Storage means" refers to the technology or device for storing all audio data, text data, and sentiment analysis results.

[0391] A "wireless communication system" is a system for wireless communication within an aircraft.

[0392] The system of the present invention receives wireless communications within an aircraft in real time, converts the voice data into text data for visualization, and further recognizes the user's emotions and displays additional information based on the emotions. This system includes a receiving means, a voice recognition means, a generating means, an emotion analysis means, a display means, and a storage means.

[0393] First, the server captures voice data in real time from the aircraft's radio communication system. As a receiving means, a device with a wireless module is used, which ensures reliable reception of radio communications within the aircraft. The received voice data is immediately stored in a buffer.

[0394] Next, the device receives the voice data sent from the server. Using wireless communication technologies such as Wi-Fi or Bluetooth, the device captures this voice data locally. This data is converted into text data by speech recognition using the Google Speech-to-Text API.

[0395] The converted text data is then parsed by a generator and categorized into flight information such as flight number, altitude, speed, etc. This analysis process uses Python scripts and other data analysis tools, and this information is then generated into visualizations that allow users to understand the aircraft's flight status at a glance.

[0396] Additionally, Microsoft Azure's Text Analytics API is used for sentiment analysis. Pilot emotions are analyzed from voice and text data to detect signs of tension or fatigue. The results of this sentiment analysis are also reflected in the visualized data, providing pilots with richer information.

[0397] All generated data, including audio data, text data, and sentiment analysis results, are stored in a database by a storage means, and this storage process is designed to be available for future analysis and validation.

[0398] The visualized data and the results of the emotion analysis are displayed on the device's display. For example, if the command "Flight 123, descend to 10,000 feet" is received, this is converted into text data in real time and the visualized data "Flight 123 has been instructed to descend to an altitude of 10,000 feet" is displayed. At the same time, if any signs of tension are shown, that information is also displayed.

[0399] An example prompt is:

[0400] "If a radio command such as 'Flight 123, descend to 10,000 feet' is received, explain how you would convert the audio data into text and visualize it. Also, explain how you would incorporate sentiment analysis results into the visualization."

[0401] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0402] Step 1:

[0403] The server receives real-time voice data from the aircraft's radio communication system. Specifically, it uses a radio module to capture radio communication within the aircraft. The input is the voice signal captured from the aircraft's radio frequency, and the output is the digitized voice data.

[0404] Step 2:

[0405] The server temporarily stores the received audio data in a buffer, which prevents data loss or delay. The input is digitized audio data, and the output is audio data stored in the buffer in memory.

[0406] Step 3:

[0407] The device receives audio data from the server. Specifically, the audio data is transferred via Wi-Fi or Bluetooth. The input is the audio data transmitted wirelessly from the server, and the output is the audio data stored in the device's temporary storage.

[0408] Step 4:

[0409] The device converts the received voice data into text data using a voice recognition method. Using the Google Speech-to-Text API, the audio file is sent to the API to obtain text data. The input is the voice data stored on the device, and the output is the text data converted from the voice data.

[0410] Step 5:

[0411] The terminal analyzes the text data using a generation means and generates visualization data of aircraft operation status. Using a Python script, it extracts information such as flight number, altitude, and speed and generates this into graphs and icons. The input is the text data generated by the speech recognition means, and the output is the visualization data.

[0412] Step 6:

[0413] The device uses emotion analysis to analyze the user's emotions. It uses Microsoft Azure's Text Analytics API to send text data to the API and obtains the emotion analysis results. The input is voice or text data, and the output is analysis data that indicates the user's emotional state.

[0414] Step 7:

[0415] The device reflects the results of the emotion analysis in the visualized data. Based on the results of the emotion analysis, alerts and attention-grabbing elements are added to the visualized data. The input is the emotion data generated by the emotion analysis means, and the output is the highlighted visualized data.

[0416] Step 8:

[0417] The device stores all generated data in a storage means. Specifically, it stores voice data, text data, and emotion analysis results in a database. The input is the visualization data and emotion analysis data, and the output is the data stored in the database.

[0418] Step 9:

[0419] The terminal displays the visualized data and emotion analysis results on a display. The pilot can visually confirm the instructions and his / her own emotional state. The input is the data obtained from the generation means and emotion analysis means, and the output is the visualized information and emotion information displayed on the display.

[0420] (Application example 2)

[0421] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0422] Conventional systems can receive aircraft radio communications in real time and visualize their content, but they are unable to provide information that takes into account the user's emotional state. This has led to the problem that pilots and security staff are unable to respond appropriately when faced with an emergency or excessive stress. The present invention aims to solve this problem by providing a system that provides advanced information that takes into account the user's emotional state.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0424] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the aircraft's status based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, emotion analysis means for analyzing the user's emotions, and alert display means for displaying an alert based on the emotion analysis result, thereby making it possible to visualize the contents of wireless communications and display an alert according to the user's emotional state.

[0425] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0426] "Speech recognition means" refers to software or hardware for converting received voice data into text data.

[0427] A "generator" is a process or device for visualizing aircraft situations based on text data.

[0428] "Display means" refers to a display or other display device for showing visualized situation data to a user.

[0429] "Storage Means" means a data storage device for recording and archiving all audio and text data.

[0430] "Emotion analysis means" refers to software or artificial intelligence models for analyzing emotions from a user's voice or text.

[0431] The "alert display means" is a display device or system for notifying the user of necessary alerts based on the emotion analysis results.

[0432] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and also recognizes the user's emotions and displays additional information based on those emotions. The specific configuration and processing of this system will be described below.

[0433] First, the server captures voice data in real time from the aircraft's radio communication system. The captured voice data is temporarily stored in a buffer. The terminal then receives the voice data sent from the server, inputs this voice data into a voice recognition means, and converts it into text data. At this stage, the "speech_recognition" library is used to convert the voice data into text data, and then emotion analysis is performed using the "transformers" library and Hugging Face's "pipeline."

[0434] The generator analyzes the converted text data and appropriately visualizes the aircraft status data, specifically classifying the instructions and information converted from the voice into flight information such as flight number, altitude, speed, etc., using appropriately designed templates and dashboards.

[0435] Next, the sentiment analysis means analyzes the text and voice data to determine the user's emotional state. This process uses the Sentiment Analysis Pipeline. The results of the sentiment analysis evaluate the user's level of tension, fatigue, etc., and based on this, the system decides whether to display an alert.

[0436] The alert display means provides visual alerts to the user as needed based on the emotion analysis results. If abnormal emotion fluctuations or signs of danger are detected, an alert is displayed immediately. This alert display is performed using devices such as smart glasses or a head-mounted display.

[0437] For example, if a security staff member radios in to say, "An important person has entered the building," the system can convert this speech into text in real time and perform sentiment analysis. If the process determines that the staff member is overstressed, an appropriate alert will be displayed on the smart glasses.

[0438] Prompt Sentence Examples

[0439] Convert the speech "An important person has entered the building" into text, run it through a sentiment analysis model, and display the results.

[0440] This allows the system of the present invention to improve the safety and efficiency of aircraft operations and security operations by combining real-time speech-to-text conversion with emotional state recognition and providing information based on these.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] The server receives voice data in real time from the aircraft's radio communication system. The received voice data is temporarily stored in a buffer. At this stage, the input is the voice data from the radio communication system, and the output is the voice data stored in the buffer.

[0444] Step 2:

[0445] The device receives the voice data sent from the server. It converts this voice data into text data using a speech recognition tool. This process uses the "speech_recognition" library. The input is the voice data in the buffer, and the output is the converted text data.

[0446] Step 3:

[0447] The generation means receives the text data obtained from the speech recognition means and analyzes the text data. During the analysis process, the text data is classified into aircraft operation information such as flight number, altitude, and speed, and appropriate visualization data is generated. The input is text data, and the output is visualization data. Templates and dashboards are used for this visualization data.

[0448] Step 4:

[0449] The emotion analysis method uses text and audio data to analyze the user's emotional state. The "transformers" library and Hugging Face's "pipeline" are used to perform emotion evaluation based on text and audio data. The input is text and audio data, and the output is the emotion analysis result.

[0450] Step 5:

[0451] Based on the emotion analysis results, the terminal displays an alert as necessary. The alert display means displays visual and auditory alerts to the user on an appropriate device (such as smart glasses or a head-mounted display) based on the emotion analysis results. The input is the emotion analysis results, and the output is the visual and auditory alerts.

[0452] Step 6:

[0453] The terminal displays the generated visualization data and emotion analysis results on a display, providing them to pilots and security staff. This allows users to grasp the aircraft's operational status and their own emotional state in real time. The input is the visualization data and emotion analysis results, and the output is the information displayed on the display.

[0454] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0455] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0456] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0457] [Third embodiment]

[0458] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0459] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0460] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0461] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0462] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0463] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0464] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0465] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0466] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0467] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0468] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0469] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0470] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0471] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0472] The terminal then receives the voice data sent from the server and converts the voice data into text data using a voice recognition means, which uses a pre-trained voice recognition model.

[0473] The text data generated by the speech recognition means is processed by a generation means, which analyzes the text data and generates visualization data representing the aircraft's current operational status and instructions, including details such as flight number, current altitude, target altitude, and descent instructions.

[0474] The terminal then retrieves the visualization data generated by the generating means and displays it to the pilot using the displaying means, which may be a display or other suitable display device.

[0475] Additionally, the server stores all audio data and text data generated by speech recognition in local storage using a storage means, making the data available for future analysis and verification.

[0476] To give a specific example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." The terminal displays this visualization data on a display, allowing the pilot to visually confirm the content of the instruction, reducing mishearing and misrecognition.

[0477] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0478] The processing flow will be explained below.

[0479] Step 1:

[0480] The server captures voice data from the aircraft's radio communication system in real time, and after receiving the voice data, stores it in a temporary buffer, allowing the radio communication content to proceed to the next processing step without loss.

[0481] Step 2:

[0482] The terminal receives voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0483] Step 3:

[0484] After the text data is generated, the generating means of the terminal receives the text data and starts analyzing it, classifying the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generating appropriate visualization data.

[0485] Step 4:

[0486] The terminal receives the generated visualization data and transmits it to the display means, which displays the generated visualization data to the pilot using a display or other suitable display device, thereby allowing the pilot to visually confirm the current operational status and instructions.

[0487] Step 5:

[0488] The server will store all audio data and text data generated by speech recognition in local storage using a storage method, ensuring that the data is available for future analysis and verification.

[0489] In this way, a series of processing steps results in a system that accurately captures the contents of radio communications and provides them visually to the pilot, thereby reducing mishearing and misinterpretation of radio communications and improving aviation safety.

[0490] Example 1

[0491] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0492] Radio communication within an aircraft is important for pilots to receive instructions in real time, but voice communication is prone to misunderstandings and mishearing. Therefore, it is necessary to improve recognition accuracy by converting voice data into text data and visualizing it. However, current systems have difficulty achieving real-time and reliable voice recognition and data visualization.

[0493] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0494] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the status of the aircraft based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, analysis means for extracting the operating status and instructions of the aircraft, and communication means for transmitting voice data from the server to the terminal. This enables real-time text conversion and visualization of voice data, reducing pilot judgment errors and ensuring safety.

[0495] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0496] "Speech recognition means" refers to a technique for converting received voice data into text data.

[0497] The "generation means" is a technology for visualizing the aircraft situation based on text data.

[0498] "Display means" means a device for displaying visualized situation data to the pilot.

[0499] "Storage Means" means any device or technology used to store all audio and text data.

[0500] The "analysis means" is a technology for extracting aircraft operating status and instruction content from text data.

[0501] "Communication means" refers to the technology for transmitting audio data from the server to the terminal.

[0502] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0503] The server captures voice data from the aircraft's radio communication system in real time. Specifically, the server acquires the voice data through a radio receiver installed on the aircraft and temporarily stores it in a buffer. For example, a high-performance radio receiver combined with dedicated data capture software can be used.

[0504] Next, the terminal receives the voice data sent from the server. At this time, the terminal converts the voice data into text data using a voice recognition means. As the voice recognition means, an existing voice recognition service such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text can be used.

[0505] The terminal receives the text data generated by speech recognition and analyzes it using an analysis means. This analysis uses a Python program or a natural language processing library (e.g., SpaCy, NLTK). The analysis means extracts the aircraft's operational status and instructions from the text data.

[0506] The terminal then generates visualization data based on the extracted information using a generation means. The visualization data includes information such as the flight number, current altitude, target altitude, and descent instructions. The generated visualization data is sent to an appropriate display device, such as a display or tablet, and displayed to the pilot. The display means can be a high-resolution display or a mobile tablet.

[0507] Furthermore, the server stores all voice data and text data generated by speech recognition in local storage (SSD or HDD) using a storage means, making the data available for future analysis and verification. For example, using a high-speed, reliable SSD allows for efficient data storage and retrieval.

[0508] As a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The device receives this voice data and converts it into text data "Flight 123, descend to 10,000 feet" using the Google Cloud Speech-to-Text API. The device then analyzes this text data using the SpaCy library and extracts information such as the flight number "Flight 123" and the intended descent altitude "10,000 feet." The device then generates visualization data based on the extracted information and displays it on the display. The pilot then checks the visualization data on the display and recognizes the descent instruction.

[0509] Example prompts to input to a generative AI model:

[0510] "Convert the audio data of the following radio communication into text data and generate a visualization. Audio data: "Flight 123, descend to 10,000 feet""

[0511] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0512] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0513] Step 1:

[0514] The server captures voice data in real time from the wireless communication system in the aircraft. Using a receiving means, it captures the voice signal transmitted from the wireless communication device and converts it into digital voice data. The input is the analog voice signal from the wireless communication, and the output is the digital voice data. This data is temporarily stored in a buffer.

[0515] Step 2:

[0516] The terminal receives the voice data sent from the server. The terminal receives the voice data from the server using a communication means and temporarily stores it in local storage. The input is the digital voice data sent from the server, and the output is the voice data stored in the local storage.

[0517] Step 3:

[0518] The device converts the received voice data into text data using a voice recognition means. Using a voice recognition means (e.g., Google Cloud Speech-to-Text API), the voice data is analyzed and converted into text format. The input is the voice data stored in local storage, and the output is the text data generated by voice recognition.

[0519] Step 4:

[0520] The terminal analyzes the generated text data using an analysis means. A natural language processing library (e.g., SpaCy, NLTK) is used to extract the aircraft's operational status and instructions from the text data. The input is the text data generated by the speech recognition means, and the output is the extracted operational status and instructions. Specifically, the terminal extracts keywords such as flight number and altitude from the text and performs analysis based on these.

[0521] Step 5:

[0522] The terminal generates visualization data using a generation means based on the extracted information. The generated visualization data includes information such as the flight number, current altitude, destination altitude, and descent instructions. The input is the flight status and instructions extracted by the analysis means, and the output is data that can be displayed visually. Specifically, the terminal generates a graphical interface based on the extracted information and structures it in an appropriate format.

[0523] Step 6:

[0524] The terminal displays the generated visualization data to the pilot using a display means. The display means may be a high-resolution display or a mobile tablet. The input is the visualization data from the generation means, and the output is the data displayed on the display device. Specific operations include controlling the layout and display of the data on the display.

[0525] Step 7:

[0526] The server uses a storage means to store all voice data and text data generated by voice recognition in local storage. The input is the captured voice data and text data, and the output is the data stored in local storage. Specifically, the data is stored in a high-speed, reliable storage device (SSD or HDD) and made available for later analysis and verification.

[0527] (Application example 1)

[0528] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0529] In current logistics centers, when workers receive work instructions via voice, efficiency can decrease due to mishearing or lack of recognition. Manual confirmation of work instructions also poses a problem, resulting in work delays. In particular, when workers work for long periods of time while connected to a wireless network, receiving continuous instructions increases the risk of overwork. We propose a new system to solve these issues and provide an efficient and accurate method of transmitting work instructions.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0531] In this invention, the server includes a receiving means for receiving wireless communications in real time, a voice recognition means for converting received voice data into text data, a generating means for visualizing a situation based on the text data, a display means for displaying the visualized situation data, and a storage means for saving all voice and text data, thereby enabling workers to receive accurate work instructions in real time through smart glasses or a head-mounted display, and to work efficiently.

[0532] "Wireless communication" is a technology for transmitting and receiving information such as voice data and text data using wireless signals.

[0533] The "receiving means" is a device or system for receiving voice data and other data transmitted by wireless communication in real time.

[0534] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0535] The "generation means" is a technology or device for analyzing a situation based on the converted text data and generating visualization data.

[0536] "Display means" refers to a technology or device for displaying the generated visualization data to the worker. This includes smart glasses and head-mounted displays.

[0537] "Storage means" refers to technology or devices for storing received voice data and generated text data.

[0538] A "generative AI model" is an artificial intelligence model that analyzes situations based on text data and visualizes work instructions.

[0539] The present invention relates to a system that allows workers at a logistics center to visually obtain work instructions in real time using smart glasses or a head-mounted display.

[0540] 1. Capture audio data

[0541] The server receives voice data in real time using a wireless communication system, specifically, captures voice data including work instructions through a wireless communication device, and temporarily stores the captured voice data in a buffer.

[0542] 2. Speech Recognition and Text Conversion

[0543] The device converts the voice data sent from the server into text data in real time using a speech recognition method. The speech recognition method uses the "speech_recognition" library. The voice data is converted into text through the Google Speech Recognition API.

[0544] 3. Analysis and visualization of text data

[0545] The generation means analyzes the converted text data and generates visualization data containing specific work instructions. Here, a generative AI model is used to convert the work instructions based on the text data into a form that is visually easy to understand. This generative AI model uses, for example, a natural language processing model for text analysis.

[0546] 4. Displaying Visualized Data

[0547] The device then displays the generated visualization data on smart glasses or a head-mounted display, allowing workers to view the visualized work instructions in real time. Examples of display methods include Google Glass or similar smart glasses.

[0548] 5. Data storage

[0549] The server uses a storage means to store all received audio data and generated text data in local storage, ensuring data availability for future analysis and verification.

[0550] Specific examples

[0551] As a concrete example, let's consider the case where an instruction is given via wireless communication within a logistics center: "Pick up Box 42 from Zone 5 and move it to Zone 10." The server receives this voice data in real time and temporarily stores it. The device uses the Google Speech Recognition API to convert this voice data into text data: "Pick up Box 42 from Zone 5 and move it to Zone 10."

[0552] Next, the generation means analyzes this text data and generates visualization data as specific work instructions, such as "Pick up Box 42 from Zone 5 and move it to Zone 10." The worker's smart glasses display this visualization data, allowing the worker to check it in real time.

[0553] Examples of prompt statements

[0554] Listening for instructions...

[0555] Instructions: Pick up Box 42 from Zone 5 and move it to Zone 10

[0556] Recognized text: Pick up Box 42 from Zone 5 and move to Zone 10

[0557] Displaying instructions on smart glasses:

[0558] {

[0559] "timestamp": "2023-10-10 14:25:30",

[0560] "instruction": "Pick up Box 42 from Zone 5 and move it to Zone 10"

[0561] }

[0562] In this way, the system of the present invention can improve work efficiency in a logistics center and increase the accuracy of the transmission of work instructions.

[0563] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0564] Step 1:

[0565] The server receives voice data in real time through a wireless communication system. Specifically, it uses a wireless communication device to capture voice data such as work instructions. This voice data is temporarily stored in a buffer. The input is a wireless communication signal, and the output is the captured voice data.

[0566] Step 2:

[0567] The device receives the voice data sent from the server and converts it into text data using a speech recognition method. This speech recognition method uses the "speech_recognition" library and converts the voice data into text data via the Google Speech Recognition API. The input is voice data and the output is text data.

[0568] Step 3:

[0569] The terminal passes the converted text data to a generation means, which analyzes the text data and generates visualization data. Specifically, the generation means uses a natural language processing model to analyze the text data. The generated visualization data includes specific work instruction content. The input is text data, and the output is visualization data.

[0570] Step 4:

[0571] The terminal displays the generated visualization data on smart glasses or a head-mounted display. Google Glass or similar smart glasses are used as the display means. The input is the visualization data, and the output is the displayed work instructions. The user can confirm and carry out the work instructions through this display.

[0572] Step 5:

[0573] The server stores the received voice data and generated text data in local storage using a storage means, ensuring data for future analysis and verification. The input is the voice data and text data, and the output is the stored data.

[0574] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0575] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and further recognizes the user's emotions and displays additional information based on these. The specific configuration and processing of the system are described below.

[0576] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0577] Next, the terminal receives the voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0578] The text data generated by the speech recognition means is processed by the generation means, which classifies the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generates appropriate visualization data.

[0579] Furthermore, the emotion engine analyzes the user's emotions based on the voice data. The results of this emotion analysis are also reflected in the visualization data, enriching the content presented to the pilot. For example, it is possible to respond by displaying an alert if there are large fluctuations in emotions.

[0580] The emotion analysis data generated by the emotion engine is also stored as text data in the storage means, ensuring data availability for future analysis and verification.

[0581] To give a concrete example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes the user's emotions, and if there are signs of tension or fatigue, for example, that information is also displayed. The terminal displays this visualization data on a display, allowing the pilot to visually confirm the instructions and their own emotional state.

[0582] In this way, the system of the present invention not only improves the recognition accuracy of radio communications on board an aircraft, but also provides information that takes into account the user's emotional state, thereby further improving safety.

[0583] The processing flow will be explained below.

[0584] Step 1:

[0585] The server captures voice data from the aircraft's radio communication system in real time and temporarily stores the captured voice data in a buffer, ensuring that the content of the radio communication is accurately used for subsequent processing.

[0586] Step 2:

[0587] The terminal receives the voice data sent from the server. The received voice data is input into the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0588] Step 3:

[0589] The generating means analyzes the text data obtained by the speech recognition means, extracts aircraft operation information, such as flight number, altitude, speed, etc., from the text data, and generates visualization data.

[0590] Step 4:

[0591] The device's emotion engine analyzes the user's emotions in real time from voice data. Emotions are evaluated using multiple indicators, including tension, fatigue, and stress. Based on the analysis results, an alert is generated if there are large fluctuations in emotions.

[0592] Step 5:

[0593] The terminal integrates the visualization data obtained by the generating means with the emotion data analyzed by the emotion engine, and transmits the integrated information to the display means for visually displaying to the pilot.

[0594] Step 6:

[0595] The display means of the terminal uses a device such as a display to display the integrated visualization data and emotion data to the pilot, thereby enabling the pilot to quickly and accurately grasp the instructions and his / her own emotional state.

[0596] Step 7:

[0597] The server stores all audio and text data, as well as sentiment analysis data, in local storage using a storage means, ensuring data is available for future analysis and validation.

[0598] To give a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal uses a voice recognition means to convert the voice data into text data such as "Flight 123, descend to 10,000 feet." The generation means analyzes the text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes signs of tension or fatigue in the pilot's voice and generates this as emotion data. The terminal's display means displays this visualization data and emotion data on a display, allowing the pilot to visually confirm the instruction and his or her own emotional state. All of this information is stored by the server and can be used for later analysis and verification.

[0599] Example 2

[0600] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0601] Radio communications on aircraft must be understood accurately and quickly, especially in emergencies. However, simply converting speech to text can lead to information omissions and misunderstandings. Furthermore, providing information without considering the psychological state of the pilot and crew can lead to further risks. Therefore, there is a need for a comprehensive information system that accurately recognizes radio communications and takes into account the user's emotional state.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0603] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting the received voice data into text data, generation means for analyzing the converted text data and generating visualized data of the aircraft situation, emotion analysis means for analyzing the user's emotions from the voice data, display means for displaying the visualized situation data and the emotion analysis results, and storage means for storing all the voice data, text data, and the emotion analysis results, thereby enabling accurate recognition of wireless communications and provision of information taking into account the user's emotions.

[0604] "Receiving means" means a device or function for receiving radio communications within an aircraft in real time.

[0605] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0606] The "generation means" is a technology or device for analyzing the converted text data and generating visualization data of the aircraft situation.

[0607] "Emotion analysis means" refers to a technique or device for analyzing a user's emotions from voice data.

[0608] The "display means" is a device or function for displaying visualized situation data and emotion analysis results.

[0609] "Storage means" refers to the technology or device for storing all audio data, text data, and sentiment analysis results.

[0610] A "wireless communication system" is a system for wireless communication within an aircraft.

[0611] The system of the present invention receives wireless communications within an aircraft in real time, converts the voice data into text data for visualization, and further recognizes the user's emotions and displays additional information based on the emotions. This system includes a receiving means, a voice recognition means, a generating means, an emotion analysis means, a display means, and a storage means.

[0612] First, the server captures voice data in real time from the aircraft's radio communication system. As a receiving means, a device with a wireless module is used, which ensures reliable reception of radio communications within the aircraft. The received voice data is immediately stored in a buffer.

[0613] Next, the device receives the voice data sent from the server. Using wireless communication technologies such as Wi-Fi or Bluetooth, the device captures this voice data locally. This data is converted into text data by speech recognition using the Google Speech-to-Text API.

[0614] The converted text data is then parsed by a generator and categorized into flight information such as flight number, altitude, speed, etc. This analysis process uses Python scripts and other data analysis tools, and this information is then generated into visualizations that allow users to understand the aircraft's flight status at a glance.

[0615] Additionally, Microsoft Azure's Text Analytics API is used for sentiment analysis. Pilot emotions are analyzed from voice and text data to detect signs of tension or fatigue. The results of this sentiment analysis are also reflected in the visualized data, providing pilots with richer information.

[0616] All generated data, including audio data, text data, and sentiment analysis results, are stored in a database by a storage means, and this storage process is designed to be available for future analysis and validation.

[0617] The visualized data and the results of the emotion analysis are displayed on the device's display. For example, if the command "Flight 123, descend to 10,000 feet" is received, this is converted into text data in real time and the visualized data "Flight 123 has been instructed to descend to an altitude of 10,000 feet" is displayed. At the same time, if any signs of tension are shown, that information is also displayed.

[0618] An example prompt is:

[0619] "If a radio command such as 'Flight 123, descend to 10,000 feet' is received, explain how you would convert the audio data into text and visualize it. Also, explain how you would incorporate sentiment analysis results into the visualization."

[0620] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0621] Step 1:

[0622] The server receives real-time voice data from the aircraft's radio communication system. Specifically, it uses a radio module to capture radio communication within the aircraft. The input is the voice signal captured from the aircraft's radio frequency, and the output is the digitized voice data.

[0623] Step 2:

[0624] The server temporarily stores the received audio data in a buffer, which prevents data loss or delay. The input is digitized audio data, and the output is audio data stored in the buffer in memory.

[0625] Step 3:

[0626] The device receives audio data from the server. Specifically, the audio data is transferred via Wi-Fi or Bluetooth. The input is the audio data transmitted wirelessly from the server, and the output is the audio data stored in the device's temporary storage.

[0627] Step 4:

[0628] The device converts the received voice data into text data using a voice recognition method. Using the Google Speech-to-Text API, the audio file is sent to the API to obtain text data. The input is the voice data stored on the device, and the output is the text data converted from the voice data.

[0629] Step 5:

[0630] The terminal analyzes the text data using a generation means and generates visualization data of aircraft operation status. Using a Python script, it extracts information such as flight number, altitude, and speed and generates this into graphs and icons. The input is the text data generated by the speech recognition means, and the output is the visualization data.

[0631] Step 6:

[0632] The device uses emotion analysis to analyze the user's emotions. It uses Microsoft Azure's Text Analytics API to send text data to the API and obtains the emotion analysis results. The input is voice or text data, and the output is analysis data that indicates the user's emotional state.

[0633] Step 7:

[0634] The device reflects the results of the emotion analysis in the visualized data. Based on the results of the emotion analysis, alerts and attention-grabbing elements are added to the visualized data. The input is the emotion data generated by the emotion analysis means, and the output is the highlighted visualized data.

[0635] Step 8:

[0636] The device stores all generated data in a storage means. Specifically, it stores voice data, text data, and emotion analysis results in a database. The input is the visualization data and emotion analysis data, and the output is the data stored in the database.

[0637] Step 9:

[0638] The terminal displays the visualized data and emotion analysis results on a display. The pilot can visually confirm the instructions and his / her own emotional state. The input is the data obtained from the generation means and emotion analysis means, and the output is the visualized information and emotion information displayed on the display.

[0639] (Application example 2)

[0640] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0641] Conventional systems can receive aircraft radio communications in real time and visualize their content, but they are unable to provide information that takes into account the user's emotional state. This has led to the problem that pilots and security staff are unable to respond appropriately when faced with an emergency or excessive stress. The present invention aims to solve this problem by providing a system that provides advanced information that takes into account the user's emotional state.

[0642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0643] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the aircraft's status based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, emotion analysis means for analyzing the user's emotions, and alert display means for displaying an alert based on the emotion analysis result, thereby making it possible to visualize the contents of wireless communications and display an alert according to the user's emotional state.

[0644] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0645] "Speech recognition means" refers to software or hardware for converting received voice data into text data.

[0646] A "generator" is a process or device for visualizing aircraft situations based on text data.

[0647] "Display means" refers to a display or other display device for showing visualized situation data to a user.

[0648] "Storage Means" means a data storage device for recording and archiving all audio and text data.

[0649] "Emotion analysis means" refers to software or artificial intelligence models for analyzing emotions from a user's voice or text.

[0650] The "alert display means" is a display device or system for notifying the user of necessary alerts based on the emotion analysis results.

[0651] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and also recognizes the user's emotions and displays additional information based on those emotions. The specific configuration and processing of this system will be described below.

[0652] First, the server captures voice data in real time from the aircraft's radio communication system. The captured voice data is temporarily stored in a buffer. The terminal then receives the voice data sent from the server, inputs this voice data into a voice recognition means, and converts it into text data. At this stage, the "speech_recognition" library is used to convert the voice data into text data, and then emotion analysis is performed using the "transformers" library and Hugging Face's "pipeline."

[0653] The generator analyzes the converted text data and appropriately visualizes the aircraft status data, specifically classifying the instructions and information converted from the voice into flight information such as flight number, altitude, speed, etc., using appropriately designed templates and dashboards.

[0654] Next, the sentiment analysis means analyzes the text and voice data to determine the user's emotional state. This process uses the Sentiment Analysis Pipeline. The results of the sentiment analysis evaluate the user's level of tension, fatigue, etc., and based on this, the system decides whether to display an alert.

[0655] The alert display means provides visual alerts to the user as needed based on the emotion analysis results. If abnormal emotion fluctuations or signs of danger are detected, an alert is displayed immediately. This alert display is performed using devices such as smart glasses or a head-mounted display.

[0656] For example, if a security staff member radios in to say, "An important person has entered the building," the system can convert this speech into text in real time and perform sentiment analysis. If the process determines that the staff member is overstressed, an appropriate alert will be displayed on the smart glasses.

[0657] Prompt Sentence Examples

[0658] Convert the speech "An important person has entered the building" into text, run it through a sentiment analysis model, and display the results.

[0659] This allows the system of the present invention to improve the safety and efficiency of aircraft operations and security operations by combining real-time speech-to-text conversion with emotional state recognition and providing information based on these.

[0660] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0661] Step 1:

[0662] The server receives voice data in real time from the aircraft's radio communication system. The received voice data is temporarily stored in a buffer. At this stage, the input is the voice data from the radio communication system, and the output is the voice data stored in the buffer.

[0663] Step 2:

[0664] The device receives the voice data sent from the server. It converts this voice data into text data using a speech recognition tool. This process uses the "speech_recognition" library. The input is the voice data in the buffer, and the output is the converted text data.

[0665] Step 3:

[0666] The generation means receives the text data obtained from the speech recognition means and analyzes the text data. During the analysis process, the text data is classified into aircraft operation information such as flight number, altitude, and speed, and appropriate visualization data is generated. The input is text data, and the output is visualization data. Templates and dashboards are used for this visualization data.

[0667] Step 4:

[0668] The emotion analysis method uses text and audio data to analyze the user's emotional state. The "transformers" library and Hugging Face's "pipeline" are used to perform emotion evaluation based on text and audio data. The input is text and audio data, and the output is the emotion analysis result.

[0669] Step 5:

[0670] Based on the emotion analysis results, the terminal displays an alert as necessary. The alert display means displays visual and auditory alerts to the user on an appropriate device (such as smart glasses or a head-mounted display) based on the emotion analysis results. The input is the emotion analysis results, and the output is the visual and auditory alerts.

[0671] Step 6:

[0672] The terminal displays the generated visualization data and emotion analysis results on a display, providing them to pilots and security staff. This allows users to grasp the aircraft's operational status and their own emotional state in real time. The input is the visualization data and emotion analysis results, and the output is the information displayed on the display.

[0673] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0674] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0675] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0676] [Fourth embodiment]

[0677] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0678] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0679] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0680] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0681] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0682] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0683] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0684] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0685] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0686] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0687] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0688] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0689] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0690] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0691] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0692] The terminal then receives the voice data sent from the server and converts the voice data into text data using a voice recognition means, which uses a pre-trained voice recognition model.

[0693] The text data generated by the speech recognition means is processed by a generation means, which analyzes the text data and generates visualization data representing the aircraft's current operational status and instructions, including details such as flight number, current altitude, target altitude, and descent instructions.

[0694] The terminal then retrieves the visualization data generated by the generating means and displays it to the pilot using the displaying means, which may be a display or other suitable display device.

[0695] Additionally, the server stores all audio data and text data generated by speech recognition in local storage using a storage means, making the data available for future analysis and verification.

[0696] To give a specific example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." The terminal displays this visualization data on a display, allowing the pilot to visually confirm the content of the instruction, reducing mishearing and misrecognition.

[0697] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0698] The processing flow will be explained below.

[0699] Step 1:

[0700] The server captures voice data from the aircraft's radio communication system in real time, and after receiving the voice data, stores it in a temporary buffer, allowing the radio communication content to proceed to the next processing step without loss.

[0701] Step 2:

[0702] The terminal receives voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0703] Step 3:

[0704] After the text data is generated, the generating means of the terminal receives the text data and starts analyzing it, classifying the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generating appropriate visualization data.

[0705] Step 4:

[0706] The terminal receives the generated visualization data and transmits it to the display means, which displays the generated visualization data to the pilot using a display or other suitable display device, thereby allowing the pilot to visually confirm the current operational status and instructions.

[0707] Step 5:

[0708] The server will store all audio data and text data generated by speech recognition in local storage using a storage method, ensuring that the data is available for future analysis and verification.

[0709] In this way, a series of processing steps results in a system that accurately captures the contents of radio communications and provides them visually to the pilot, thereby reducing mishearing and misinterpretation of radio communications and improving aviation safety.

[0710] Example 1

[0711] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0712] Radio communication within an aircraft is important for pilots to receive instructions in real time, but voice communication is prone to misunderstandings and mishearing. Therefore, it is necessary to improve recognition accuracy by converting voice data into text data and visualizing it. However, current systems have difficulty achieving real-time and reliable voice recognition and data visualization.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0714] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the status of the aircraft based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, analysis means for extracting the operating status and instructions of the aircraft, and communication means for transmitting voice data from the server to the terminal. This enables real-time text conversion and visualization of voice data, reducing pilot judgment errors and ensuring safety.

[0715] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0716] "Speech recognition means" refers to a technique for converting received voice data into text data.

[0717] The "generation means" is a technology for visualizing the aircraft situation based on text data.

[0718] "Display means" means a device for displaying visualized situation data to the pilot.

[0719] "Storage Means" means any device or technology used to store all audio and text data.

[0720] The "analysis means" is a technology for extracting aircraft operating status and instruction content from text data.

[0721] "Communication means" refers to the technology for transmitting audio data from the server to the terminal.

[0722] The system of the present invention receives radio communications in real time inside an aircraft, converts the voice data into text data, visualizes it, and displays it to the pilot. The specific configuration and processing of the system are described below.

[0723] The server captures voice data from the aircraft's radio communication system in real time. Specifically, the server acquires the voice data through a radio receiver installed on the aircraft and temporarily stores it in a buffer. For example, a high-performance radio receiver combined with dedicated data capture software can be used.

[0724] Next, the terminal receives the voice data sent from the server. At this time, the terminal converts the voice data into text data using a voice recognition means. As the voice recognition means, an existing voice recognition service such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text can be used.

[0725] The terminal receives the text data generated by speech recognition and analyzes it using an analysis means. This analysis uses a Python program or a natural language processing library (e.g., SpaCy, NLTK). The analysis means extracts the aircraft's operational status and instructions from the text data.

[0726] The terminal then generates visualization data based on the extracted information using a generation means. The visualization data includes information such as the flight number, current altitude, target altitude, and descent instructions. The generated visualization data is sent to an appropriate display device, such as a display or tablet, and displayed to the pilot. The display means can be a high-resolution display or a mobile tablet.

[0727] Furthermore, the server stores all voice data and text data generated by speech recognition in local storage (SSD or HDD) using a storage means, making the data available for future analysis and verification. For example, using a high-speed, reliable SSD allows for efficient data storage and retrieval.

[0728] As a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The device receives this voice data and converts it into text data "Flight 123, descend to 10,000 feet" using the Google Cloud Speech-to-Text API. The device then analyzes this text data using the SpaCy library and extracts information such as the flight number "Flight 123" and the intended descent altitude "10,000 feet." The device then generates visualization data based on the extracted information and displays it on the display. The pilot then checks the visualization data on the display and recognizes the descent instruction.

[0729] Example prompts to input to a generative AI model:

[0730] "Convert the audio data of the following radio communication into text data and generate a visualization. Audio data: "Flight 123, descend to 10,000 feet""

[0731] In this way, the system of the present invention can improve the accuracy of recognizing radio communications inside an aircraft, reduce pilot errors in judgment, and ensure safety.

[0732] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0733] Step 1:

[0734] The server captures voice data in real time from the wireless communication system in the aircraft. Using a receiving means, it captures the voice signal transmitted from the wireless communication device and converts it into digital voice data. The input is the analog voice signal from the wireless communication, and the output is the digital voice data. This data is temporarily stored in a buffer.

[0735] Step 2:

[0736] The terminal receives the voice data sent from the server. The terminal receives the voice data from the server using a communication means and temporarily stores it in local storage. The input is the digital voice data sent from the server, and the output is the voice data stored in the local storage.

[0737] Step 3:

[0738] The device converts the received voice data into text data using a voice recognition means. Using a voice recognition means (e.g., Google Cloud Speech-to-Text API), the voice data is analyzed and converted into text format. The input is the voice data stored in local storage, and the output is the text data generated by voice recognition.

[0739] Step 4:

[0740] The terminal analyzes the generated text data using an analysis means. A natural language processing library (e.g., SpaCy, NLTK) is used to extract the aircraft's operational status and instructions from the text data. The input is the text data generated by the speech recognition means, and the output is the extracted operational status and instructions. Specifically, the terminal extracts keywords such as flight number and altitude from the text and performs analysis based on these.

[0741] Step 5:

[0742] The terminal generates visualization data using a generation means based on the extracted information. The generated visualization data includes information such as the flight number, current altitude, destination altitude, and descent instructions. The input is the flight status and instructions extracted by the analysis means, and the output is data that can be displayed visually. Specifically, the terminal generates a graphical interface based on the extracted information and structures it in an appropriate format.

[0743] Step 6:

[0744] The terminal displays the generated visualization data to the pilot using a display means. The display means may be a high-resolution display or a mobile tablet. The input is the visualization data from the generation means, and the output is the data displayed on the display device. Specific operations include controlling the layout and display of the data on the display.

[0745] Step 7:

[0746] The server uses a storage means to store all voice data and text data generated by voice recognition in local storage. The input is the captured voice data and text data, and the output is the data stored in local storage. Specifically, the data is stored in a high-speed, reliable storage device (SSD or HDD) and made available for later analysis and verification.

[0747] (Application example 1)

[0748] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0749] In current logistics centers, when workers receive work instructions via voice, efficiency can decrease due to mishearing or lack of recognition. Manual confirmation of work instructions also poses a problem, resulting in work delays. In particular, when workers work for long periods of time while connected to a wireless network, receiving continuous instructions increases the risk of overwork. We propose a new system to solve these issues and provide an efficient and accurate method of transmitting work instructions.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0751] In this invention, the server includes a receiving means for receiving wireless communications in real time, a voice recognition means for converting received voice data into text data, a generating means for visualizing a situation based on the text data, a display means for displaying the visualized situation data, and a storage means for saving all voice and text data, thereby enabling workers to receive accurate work instructions in real time through smart glasses or a head-mounted display, and to work efficiently.

[0752] "Wireless communication" is a technology for transmitting and receiving information such as voice data and text data using wireless signals.

[0753] The "receiving means" is a device or system for receiving voice data and other data transmitted by wireless communication in real time.

[0754] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0755] The "generation means" is a technology or device for analyzing a situation based on the converted text data and generating visualization data.

[0756] "Display means" refers to a technology or device for displaying the generated visualization data to the worker. This includes smart glasses and head-mounted displays.

[0757] "Storage means" refers to technology or devices for storing received voice data and generated text data.

[0758] A "generative AI model" is an artificial intelligence model that analyzes situations based on text data and visualizes work instructions.

[0759] The present invention relates to a system that allows workers at a logistics center to visually obtain work instructions in real time using smart glasses or a head-mounted display.

[0760] 1. Capture audio data

[0761] The server receives voice data in real time using a wireless communication system, specifically, captures voice data including work instructions through a wireless communication device, and temporarily stores the captured voice data in a buffer.

[0762] 2. Speech Recognition and Text Conversion

[0763] The device converts the voice data sent from the server into text data in real time using a speech recognition method. The speech recognition method uses the "speech_recognition" library. The voice data is converted into text through the Google Speech Recognition API.

[0764] 3. Analysis and visualization of text data

[0765] The generation means analyzes the converted text data and generates visualization data containing specific work instructions. Here, a generative AI model is used to convert the work instructions based on the text data into a form that is visually easy to understand. This generative AI model uses, for example, a natural language processing model for text analysis.

[0766] 4. Displaying Visualized Data

[0767] The device then displays the generated visualization data on smart glasses or a head-mounted display, allowing workers to view the visualized work instructions in real time. Examples of display methods include Google Glass or similar smart glasses.

[0768] 5. Data storage

[0769] The server uses a storage means to store all received audio data and generated text data in local storage, ensuring data availability for future analysis and verification.

[0770] Specific examples

[0771] As a concrete example, let's consider the case where an instruction is given via wireless communication within a logistics center: "Pick up Box 42 from Zone 5 and move it to Zone 10." The server receives this voice data in real time and temporarily stores it. The device uses the Google Speech Recognition API to convert this voice data into text data: "Pick up Box 42 from Zone 5 and move it to Zone 10."

[0772] Next, the generation means analyzes this text data and generates visualization data as specific work instructions, such as "Pick up Box 42 from Zone 5 and move it to Zone 10." The worker's smart glasses display this visualization data, allowing the worker to check it in real time.

[0773] Examples of prompt statements

[0774] Listening for instructions...

[0775] Instructions: Pick up Box 42 from Zone 5 and move it to Zone 10

[0776] Recognized text: Pick up Box 42 from Zone 5 and move to Zone 10

[0777] Displaying instructions on smart glasses:

[0778] {

[0779] "timestamp": "2023-10-10 14:25:30",

[0780] "instruction": "Pick up Box 42 from Zone 5 and move it to Zone 10"

[0781] }

[0782] In this way, the system of the present invention can improve work efficiency in a logistics center and increase the accuracy of the transmission of work instructions.

[0783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0784] Step 1:

[0785] The server receives voice data in real time through a wireless communication system. Specifically, it uses a wireless communication device to capture voice data such as work instructions. This voice data is temporarily stored in a buffer. The input is a wireless communication signal, and the output is the captured voice data.

[0786] Step 2:

[0787] The device receives the voice data sent from the server and converts it into text data using a speech recognition method. This speech recognition method uses the "speech_recognition" library and converts the voice data into text data via the Google Speech Recognition API. The input is voice data and the output is text data.

[0788] Step 3:

[0789] The terminal passes the converted text data to a generation means, which analyzes the text data and generates visualization data. Specifically, the generation means uses a natural language processing model to analyze the text data. The generated visualization data includes specific work instruction content. The input is text data, and the output is visualization data.

[0790] Step 4:

[0791] The terminal displays the generated visualization data on smart glasses or a head-mounted display. Google Glass or similar smart glasses are used as the display means. The input is the visualization data, and the output is the displayed work instructions. The user can confirm and carry out the work instructions through this display.

[0792] Step 5:

[0793] The server stores the received voice data and generated text data in local storage using a storage means, ensuring data for future analysis and verification. The input is the voice data and text data, and the output is the stored data.

[0794] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0795] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and further recognizes the user's emotions and displays additional information based on these. The specific configuration and processing of the system are described below.

[0796] First, the server captures voice data from the aircraft's radio communication system in real time, and the captured voice data is temporarily stored in a buffer.

[0797] Next, the terminal receives the voice data sent from the server. This received voice data is input to the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0798] The text data generated by the speech recognition means is processed by the generation means, which classifies the text content into aircraft operation information, such as flight number, altitude, speed, etc., and generates appropriate visualization data.

[0799] Furthermore, the emotion engine analyzes the user's emotions based on the voice data. The results of this emotion analysis are also reflected in the visualization data, enriching the content presented to the pilot. For example, it is possible to respond by displaying an alert if there are large fluctuations in emotions.

[0800] The emotion analysis data generated by the emotion engine is also stored as text data in the storage means, ensuring data availability for future analysis and verification.

[0801] To give a concrete example, when the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal converts this voice data into text data "Flight 123, descend to 10,000 feet" using a voice recognition means. The generation means analyzes this text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes the user's emotions, and if there are signs of tension or fatigue, for example, that information is also displayed. The terminal displays this visualization data on a display, allowing the pilot to visually confirm the instructions and their own emotional state.

[0802] In this way, the system of the present invention not only improves the recognition accuracy of radio communications on board an aircraft, but also provides information that takes into account the user's emotional state, thereby further improving safety.

[0803] The processing flow will be explained below.

[0804] Step 1:

[0805] The server captures voice data from the aircraft's radio communication system in real time and temporarily stores the captured voice data in a buffer, ensuring that the content of the radio communication is accurately used for subsequent processing.

[0806] Step 2:

[0807] The terminal receives the voice data sent from the server. The received voice data is input into the voice recognition means, which converts the voice data into text data. The voice recognition means uses a pre-trained model to perform voice-to-text conversion with high accuracy.

[0808] Step 3:

[0809] The generating means analyzes the text data obtained by the speech recognition means, extracts aircraft operation information, such as flight number, altitude, speed, etc., from the text data, and generates visualization data.

[0810] Step 4:

[0811] The device's emotion engine analyzes the user's emotions in real time from voice data. Emotions are evaluated using multiple indicators, including tension, fatigue, and stress. Based on the analysis results, an alert is generated if there are large fluctuations in emotions.

[0812] Step 5:

[0813] The terminal integrates the visualization data obtained by the generating means with the emotion data analyzed by the emotion engine, and transmits the integrated information to the display means for visually displaying to the pilot.

[0814] Step 6:

[0815] The display means of the terminal uses a device such as a display to display the integrated visualization data and emotion data to the pilot, thereby enabling the pilot to quickly and accurately grasp the instructions and his / her own emotional state.

[0816] Step 7:

[0817] The server stores all audio and text data, as well as sentiment analysis data, in local storage using a storage means, ensuring data is available for future analysis and validation.

[0818] To give a specific example, if the instruction "Flight 123, descend to 10,000 feet" is received via radio communication, the server receives this voice data in real time and stores it in a buffer. The terminal uses a voice recognition means to convert the voice data into text data such as "Flight 123, descend to 10,000 feet." The generation means analyzes the text data and generates visualization data such as "Flight 123 has been instructed to descend to an altitude of 10,000 feet." At the same time, the emotion engine analyzes signs of tension or fatigue in the pilot's voice and generates this as emotion data. The terminal's display means displays this visualization data and emotion data on a display, allowing the pilot to visually confirm the instruction and his or her own emotional state. All of this information is stored by the server and can be used for later analysis and verification.

[0819] Example 2

[0820] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0821] Radio communications on aircraft must be understood accurately and quickly, especially in emergencies. However, simply converting speech to text can lead to information omissions and misunderstandings. Furthermore, providing information without considering the psychological state of the pilot and crew can lead to further risks. Therefore, there is a need for a comprehensive information system that accurately recognizes radio communications and takes into account the user's emotional state.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0823] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting the received voice data into text data, generation means for analyzing the converted text data and generating visualized data of the aircraft situation, emotion analysis means for analyzing the user's emotions from the voice data, display means for displaying the visualized situation data and the emotion analysis results, and storage means for storing all the voice data, text data, and the emotion analysis results, thereby enabling accurate recognition of wireless communications and provision of information taking into account the user's emotions.

[0824] "Receiving means" means a device or function for receiving radio communications within an aircraft in real time.

[0825] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0826] The "generation means" is a technology or device for analyzing the converted text data and generating visualization data of the aircraft situation.

[0827] "Emotion analysis means" refers to a technique or device for analyzing a user's emotions from voice data.

[0828] The "display means" is a device or function for displaying visualized situation data and emotion analysis results.

[0829] "Storage means" refers to the technology or device for storing all audio data, text data, and sentiment analysis results.

[0830] A "wireless communication system" is a system for wireless communication within an aircraft.

[0831] The system of the present invention receives wireless communications within an aircraft in real time, converts the voice data into text data for visualization, and further recognizes the user's emotions and displays additional information based on the emotions. This system includes a receiving means, a voice recognition means, a generating means, an emotion analysis means, a display means, and a storage means.

[0832] First, the server captures voice data in real time from the aircraft's radio communication system. As a receiving means, a device with a wireless module is used, which ensures reliable reception of radio communications within the aircraft. The received voice data is immediately stored in a buffer.

[0833] Next, the device receives the voice data sent from the server. Using wireless communication technologies such as Wi-Fi or Bluetooth, the device captures this voice data locally. This data is converted into text data by speech recognition using the Google Speech-to-Text API.

[0834] The converted text data is then parsed by a generator and categorized into flight information such as flight number, altitude, speed, etc. This analysis process uses Python scripts and other data analysis tools, and this information is then generated into visualizations that allow users to understand the aircraft's flight status at a glance.

[0835] Additionally, Microsoft Azure's Text Analytics API is used for sentiment analysis. Pilot emotions are analyzed from voice and text data to detect signs of tension or fatigue. The results of this sentiment analysis are also reflected in the visualized data, providing pilots with richer information.

[0836] All generated data, including audio data, text data, and sentiment analysis results, are stored in a database by a storage means, and this storage process is designed to be available for future analysis and validation.

[0837] The visualized data and the results of the emotion analysis are displayed on the device's display. For example, if the command "Flight 123, descend to 10,000 feet" is received, this is converted into text data in real time and the visualized data "Flight 123 has been instructed to descend to an altitude of 10,000 feet" is displayed. At the same time, if any signs of tension are shown, that information is also displayed.

[0838] An example prompt is:

[0839] "If a radio command such as 'Flight 123, descend to 10,000 feet' is received, explain how you would convert the audio data into text and visualize it. Also, explain how you would incorporate sentiment analysis results into the visualization."

[0840] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0841] Step 1:

[0842] The server receives real-time voice data from the aircraft's radio communication system. Specifically, it uses a radio module to capture radio communication within the aircraft. The input is the voice signal captured from the aircraft's radio frequency, and the output is the digitized voice data.

[0843] Step 2:

[0844] The server temporarily stores the received audio data in a buffer, which prevents data loss or delay. The input is digitized audio data, and the output is audio data stored in the buffer in memory.

[0845] Step 3:

[0846] The device receives audio data from the server. Specifically, the audio data is transferred via Wi-Fi or Bluetooth. The input is the audio data transmitted wirelessly from the server, and the output is the audio data stored in the device's temporary storage.

[0847] Step 4:

[0848] The device converts the received voice data into text data using a voice recognition method. Using the Google Speech-to-Text API, the audio file is sent to the API to obtain text data. The input is the voice data stored on the device, and the output is the text data converted from the voice data.

[0849] Step 5:

[0850] The terminal analyzes the text data using a generation means and generates visualization data of aircraft operation status. Using a Python script, it extracts information such as flight number, altitude, and speed and generates this into graphs and icons. The input is the text data generated by the speech recognition means, and the output is the visualization data.

[0851] Step 6:

[0852] The device uses emotion analysis to analyze the user's emotions. It uses Microsoft Azure's Text Analytics API to send text data to the API and obtains the emotion analysis results. The input is voice or text data, and the output is analysis data that indicates the user's emotional state.

[0853] Step 7:

[0854] The device reflects the results of the emotion analysis in the visualized data. Based on the results of the emotion analysis, alerts and attention-grabbing elements are added to the visualized data. The input is the emotion data generated by the emotion analysis means, and the output is the highlighted visualized data.

[0855] Step 8:

[0856] The device stores all generated data in a storage means. Specifically, it stores voice data, text data, and emotion analysis results in a database. The input is the visualization data and emotion analysis data, and the output is the data stored in the database.

[0857] Step 9:

[0858] The terminal displays the visualized data and emotion analysis results on a display. The pilot can visually confirm the instructions and his / her own emotional state. The input is the data obtained from the generation means and emotion analysis means, and the output is the visualized information and emotion information displayed on the display.

[0859] (Application example 2)

[0860] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0861] Conventional systems can receive aircraft radio communications in real time and visualize their content, but they are unable to provide information that takes into account the user's emotional state. This has led to the problem that pilots and security staff are unable to respond appropriately when faced with an emergency or excessive stress. The present invention aims to solve this problem by providing a system that provides advanced information that takes into account the user's emotional state.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0863] In this invention, the server includes receiving means for receiving wireless communications within the aircraft in real time, voice recognition means for converting received voice data into text data, generation means for visualizing the aircraft's status based on the text data, display means for displaying the visualized status data, storage means for saving all voice and text data, emotion analysis means for analyzing the user's emotions, and alert display means for displaying an alert based on the emotion analysis result, thereby making it possible to visualize the contents of wireless communications and display an alert according to the user's emotional state.

[0864] "Receiving means" refers to a device for receiving radio communications within an aircraft in real time.

[0865] "Speech recognition means" refers to software or hardware for converting received voice data into text data.

[0866] A "generator" is a process or device for visualizing aircraft situations based on text data.

[0867] "Display means" refers to a display or other display device for showing visualized situation data to a user.

[0868] "Storage Means" means a data storage device for recording and archiving all audio and text data.

[0869] "Emotion analysis means" refers to software or artificial intelligence models for analyzing emotions from a user's voice or text.

[0870] The "alert display means" is a display device or system for notifying the user of necessary alerts based on the emotion analysis results.

[0871] The system of the present invention receives radio communications in real time on board an aircraft, converts the voice data into text data and visualizes it, and also recognizes the user's emotions and displays additional information based on those emotions. The specific configuration and processing of this system will be described below.

[0872] First, the server captures voice data in real time from the aircraft's radio communication system. The captured voice data is temporarily stored in a buffer. The terminal then receives the voice data sent from the server, inputs this voice data into a voice recognition means, and converts it into text data. At this stage, the "speech_recognition" library is used to convert the voice data into text data, and then emotion analysis is performed using the "transformers" library and Hugging Face's "pipeline."

[0873] The generator analyzes the converted text data and appropriately visualizes the aircraft status data, specifically classifying the instructions and information converted from the voice into flight information such as flight number, altitude, speed, etc., using appropriately designed templates and dashboards.

[0874] Next, the sentiment analysis means analyzes the text and voice data to determine the user's emotional state. This process uses the Sentiment Analysis Pipeline. The results of the sentiment analysis evaluate the user's level of tension, fatigue, etc., and based on this, the system decides whether to display an alert.

[0875] The alert display means provides visual alerts to the user as needed based on the emotion analysis results. If abnormal emotion fluctuations or signs of danger are detected, an alert is displayed immediately. This alert display is performed using devices such as smart glasses or a head-mounted display.

[0876] For example, if a security staff member radios in to say, "An important person has entered the building," the system can convert this speech into text in real time and perform sentiment analysis. If the process determines that the staff member is overstressed, an appropriate alert will be displayed on the smart glasses.

[0877] Prompt Sentence Examples

[0878] Convert the speech "An important person has entered the building" into text, run it through a sentiment analysis model, and display the results.

[0879] This allows the system of the present invention to improve the safety and efficiency of aircraft operations and security operations by combining real-time speech-to-text conversion with emotional state recognition and providing information based on these.

[0880] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0881] Step 1:

[0882] The server receives voice data in real time from the aircraft's radio communication system. The received voice data is temporarily stored in a buffer. At this stage, the input is the voice data from the radio communication system, and the output is the voice data stored in the buffer.

[0883] Step 2:

[0884] The device receives the voice data sent from the server. It converts this voice data into text data using a speech recognition tool. This process uses the "speech_recognition" library. The input is the voice data in the buffer, and the output is the converted text data.

[0885] Step 3:

[0886] The generation means receives the text data obtained from the speech recognition means and analyzes the text data. During the analysis process, the text data is classified into aircraft operation information such as flight number, altitude, and speed, and appropriate visualization data is generated. The input is text data, and the output is visualization data. Templates and dashboards are used for this visualization data.

[0887] Step 4:

[0888] The emotion analysis method uses text and audio data to analyze the user's emotional state. The "transformers" library and Hugging Face's "pipeline" are used to perform emotion evaluation based on text and audio data. The input is text and audio data, and the output is the emotion analysis result.

[0889] Step 5:

[0890] Based on the emotion analysis results, the terminal displays an alert as necessary. The alert display means displays visual and auditory alerts to the user on an appropriate device (such as smart glasses or a head-mounted display) based on the emotion analysis results. The input is the emotion analysis results, and the output is the visual and auditory alerts.

[0891] Step 6:

[0892] The terminal displays the generated visualization data and emotion analysis results on a display, providing them to pilots and security staff. This allows users to grasp the aircraft's operational status and their own emotional state in real time. The input is the visualization data and emotion analysis results, and the output is the information displayed on the display.

[0893] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0894] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0895] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[0896] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0897] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0898] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0899] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0900] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0901] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0902] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0903] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0904] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[0905] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0906] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0907] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0908] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0909] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0910] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0911] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0912] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0913] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0914] The following is further disclosed regarding the above embodiment.

[0915] (Claim 1)

[0916] receiving means for receiving radio communications within the aircraft in real time;

[0917] speech recognition means for converting received voice data into text data;

[0918] generating means for visualizing the aircraft situation based on the text data;

[0919] display means for displaying the visualized situation data;

[0920] storage means for storing all audio and text data;

[0921] A system including:

[0922] (Claim 2)

[0923] 10. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

[0924] (Claim 3)

[0925] 2. The system according to claim 1, wherein the generating means analyzes the text data and generates visualization data of the aircraft status.

[0926] "Example 1"

[0927] (Claim 1)

[0928] receiving means for receiving radio communications within the aircraft in real time;

[0929] speech recognition means for converting received voice data into text data;

[0930] generating means for visualizing the aircraft situation based on the text data;

[0931] display means for displaying the visualized situation data;

[0932] storage means for storing all audio and text data;

[0933] an analysis means for extracting the operational status and instructions of the aircraft;

[0934] A system including a communication means for transmitting voice data from a server to a terminal.

[0935] (Claim 2)

[0936] 10. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

[0937] (Claim 3)

[0938] 2. The system according to claim 1, wherein the generating means analyzes the text data and generates visualization data of the aircraft status.

[0939] "Application Example 1"

[0940] (Claim 1)

[0941] receiving means for receiving wireless communications in real time;

[0942] speech recognition means for converting received voice data into text data;

[0943] generating means for visualizing a situation based on text data;

[0944] display means for displaying the visualized situation data;

[0945] storage means for storing all audio and text data;

[0946] A system including:

[0947] (Claim 2)

[0948] 10. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

[0949] (Claim 3)

[0950] 2. The system according to claim 1, wherein the generating means analyzes the text data and generates the situation as visualization data.

[0951] (Claim 4)

[0952] 10. The system of claim 1, further comprising smart glasses or a head-mounted display for analyzing the voice data and text data and displaying work instructions to the worker in real time.

[0953] (Claim 5)

[0954] The system of claim 4, which uses a generative AI model to generate work instruction content based on visualization data.

[0955] "Example 2: Combining Emotion Engines"

[0956] (Claim 1)

[0957] receiving means for receiving radio communications within the aircraft in real time;

[0958] speech recognition means for converting received voice data into text data;

[0959] a generating means for analyzing the converted text data and generating visualization data of the aircraft status;

[0960] emotion analysis means for analyzing user emotions from voice data;

[0961] a display means for displaying the visualized situation data and the emotion analysis result;

[0962] storage means for storing all audio data, text data, and sentiment analysis results;

[0963] A system including:

[0964] (Claim 2)

[0965] 10. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

[0966] (Claim 3)

[0967] 2. The system according to claim 1, wherein the generating means analyzes the text data and generates visualization data of the aircraft status.

[0968] "Application example 2 when combining emotion engines"

[0969] (Claim 1)

[0970] receiving means for receiving radio communications within the aircraft in real time;

[0971] speech recognition means for converting received voice data into text data;

[0972] generating means for visualizing the aircraft situation based on the text data;

[0973] display means for displaying the visualized situation data;

[0974] storage means for storing all audio and text data;

[0975] emotion analysis means for analyzing the emotion of a user;

[0976] an alert display means for displaying an alert based on the emotion analysis result;

[0977] A system including:

[0978] (Claim 2)

[0979] 10. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

[0980] (Claim 3)

[0981] 2. The system according to claim 1, wherein the generating means analyzes the text data and generates visualization data of the aircraft status. [Explanation of symbols]

[0982] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. receiving means for receiving radio communications within the aircraft in real time; speech recognition means for converting received voice data into text data; generating means for visualizing the aircraft situation based on the text data; display means for displaying the visualized situation data; storage means for storing all audio and text data; A system including:

2. 2. The system of claim 1, wherein the receiving means captures audio data from a wireless communication system.

3. 2. The system according to claim 1, wherein the generating means analyzes the text data and generates the aircraft status as visualization data.

Images