system

Abstract

Provide a system. A method for receiving video data from an unmanned aerial vehicle includes: means for transmitting the received video data to an analysis center; A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes; a means for categorizing the obtained information and storing it in a database; A means for generating and storing reports based on the classified data; A system including a means for providing generated reports in a user-friendly format.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, traffic volume surveys are often conducted manually, which requires enormous costs, time, and a heavy burden on human resources. Furthermore, manual surveys have limitations on the accuracy and quantity of data they can produce, and consistency is difficult to ensure. Furthermore, it is difficult to collect and analyze wide-area traffic data in real time. This makes it difficult to meet the demand for highly accurate and timely data in urban planning and traffic regulation formulation. [Means for solving the problem]

[0005] The present invention provides a system for analyzing traffic information using video data from unmanned aerial vehicles and generating specific reports. The system includes a means for receiving video data from the unmanned aerial vehicle, a means for transmitting the received video data to an analysis center, a means for analyzing the video data at the analysis center using generative artificial intelligence to obtain traffic volume, vehicle type, and pedestrian attribute information, a means for categorizing the obtained information and storing it in a database, a means for generating and storing reports based on the categorized data, and a means for providing the generated reports in a user-friendly format. This enables the acquisition of wide-ranging and detailed visual data in real time, enabling highly accurate traffic data analysis and improving the immediacy and consistency of data for urban planning and traffic regulation formulation. Furthermore, users can instantly obtain easy-to-use reports, thereby reducing human resources and realizing efficient data processing.

[0006] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and acquires video data from the sky.

[0007] "Video data" refers to video footage and still image data taken from the sky by unmanned aerial vehicles.

[0008] An "analysis center" is a facility or system that receives video data and analyzes it using generative artificial intelligence.

[0009] "Generative artificial intelligence" refers to algorithms that use machine learning and deep learning techniques to perform data analysis and pattern recognition.

[0010] "Traffic volume" is data that represents the number of vehicles passing through a specific road or area within a certain period of time.

[0011] "Vehicle type" is a classification that indicates the type of vehicle that travels on the road (for example, passenger car, truck, bus, etc.).

[0012] "Attribute information of pedestrians" is data including personal characteristics of pedestrians, such as gender and age, extracted from video data.

[0013] "Categorizing" means dividing the analyzed data based on its content or attributes.

[0014] "Storing in a database" means systematically managing analysis results and classified data and accumulating them in a reusable form.

[0015] A "report" refers to a collection of documents and graphs that summarize analyzed data in a visually easy-to-understand format.

[0016] "User-friendly format" means a presentation or interface designed to be easily accessible and understandable by users. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0039] composition

[0040] 1. Unmanned aerial vehicle (HAPS)

[0041] HAPS flies overhead and captures video data of a specific area in real time.

[0042] 2. Server

[0043] The server has the function of receiving video data transmitted from HAPS.

[0044] The server sends the received video data to an analysis center.

[0045] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0046] The server generates and stores reports based on the classified data.

[0047] 3. Analysis Center

[0048] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0049] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0050] 4. Terminal

[0051] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0052] operation

[0053] 1. The server receives the video from HAPS

[0054] Every hour, the server receives real-time video data transmitted by HAPS, allowing it to obtain video data for a specific intersection or a wide area of ​​a city.

[0055] 2. The server sends the data to the analysis center

[0056] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0057] 3. The server receives, classifies, and saves the analysis results

[0058] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) and stores the classified data in a database.

[0059] 4. The server generates the report

[0060] Using the classified data, the server automatically generates reports detailing, for example, the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0061] 5. Viewing reports on your device

[0062] Users can access the interface using their devices to view reports generated from the server's database. Users can select reports for specific dates, times, and locations, and check and analyze traffic information.

[0063] Specific examples

[0064] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection over several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time in the form of a report.

[0065] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0069] Step 2:

[0070] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0071] Step 3:

[0072] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0073] Step 4:

[0074] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[0075] Step 5:

[0076] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[0077] Step 6:

[0078] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[0079] Step 7:

[0080] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[0081] Step 8:

[0082] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[0083] Step 9:

[0084] The user can view the report they have selected. The user can check the details of the selected report and obtain the information they need. This allows the user to easily use traffic analysis data for a specific time period or area.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional traffic information collection systems have struggled to collect and analyze a wide range of data in real time and provide it in a user-friendly report format. Furthermore, classifying various data and generating reports requires a huge amount of time and effort, making them inadequate for urban planning and traffic management, which require rapid response. Furthermore, displaying analysis results makes it difficult to provide detailed visual data tailored to specific points in time and locations.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, means for converting the format of the received video data, means for receiving analysis results from the analysis center and categorizing them into data categories, means for indexing the data stored in the database, means for generating reports based on templates, and means for accessing the reports from a terminal and displaying them at a specific date, time, or location. This enables the collection and analysis of wide-ranging and detailed traffic information in real time and the rapid provision of such information in a user-friendly format.

[0090] An "unmanned aerial vehicle" is an aircraft that flies under remote or automatic control and photographs and collects data over a specific area.

[0091] "Video data" refers to data containing visual information captured by unmanned aerial vehicles, capturing traffic conditions and people's movements.

[0092] An "analysis center" is a facility or system that analyzes received video data and extracts information such as traffic volume, vehicle types, and pedestrian attributes.

[0093] "Generative artificial intelligence" is an artificial intelligence model that has the ability to perform analysis of data and generate specific results.

[0094] "Classification by category" means dividing the acquired information based on specific attributes, such as traffic volume, vehicle type, and pedestrian attribute information.

[0095] A "database" is an information system for storing and managing classified data, enabling efficient searching and use of the data.

[0096] A "report" is a document or file that organizes analyzed data and presents it in a visually easy-to-understand format.

[0097] "User-friendly" means that the system or interface is designed and formatted to be easy for users to use.

[0098] "Format conversion" refers to the process of changing data into a format that is easy for the analysis center to process.

[0099] "Indexing" means adding identifiers or keys to data in a database to improve search efficiency.

[0100] A "template" is a predefined format or structure used to generate a report.

[0101] A "terminal" is a device used by a user to access the system and display data, examples of which include a PC or smartphone.

[0102] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle and automatically generates user-friendly reports. This system is configured as follows.

[0103] unmanned aerial vehicle

[0104] An unmanned aerial vehicle (HAPS) flies overhead and captures specific areas in real time. For example, a HAPS can capture images of major intersections in a city and send the video data to a server on the ground. The HAPS's flight altitude and camera resolution are set according to the situation, allowing it to capture detailed traffic conditions.

[0105] server

[0106] The server has the following functions:

[0107] 1. Video reception:

[0108] The server receives the video data sent from the HAPS via a monitoring port and stores it in temporary storage in real time. HTTP or FTP is used as the communication protocol.

[0109] 2. Data transmission:

[0110] The received video data is first converted into a format before being sent to the analysis center. For example, JPEG format images are converted into MP4 format videos and then sent to the analysis center.

[0111] 3. Receiving and classifying analysis results:

[0112] The analysis center returns the analysis results data, which includes information on traffic volume, vehicle types, and pedestrian attributes. The server categorizes this data and stores it in a database.

[0113] 4. Report Generation:

[0114] The server uses the classified data to automatically generate reports based on the specified template, which are saved in PDF or HTML format.

[0115] Analysis Center

[0116] The analysis center has the following functions:

[0117] 1. Data Analysis:

[0118] The received video data is analyzed using generative artificial intelligence (AI model), which calculates traffic volume, classifies vehicle types, and estimates the age and gender of pedestrians.

[0119] Terminal

[0120] Users access reports using a device (PC or smartphone).

[0121] 1. Access the interface:

[0122] Users access the server interface through a web browser or a dedicated app.

[0123] 2. Report Viewing:

[0124] Users can select reports for specific dates, times, and locations to view detailed traffic information, including, for example, traffic volume fluctuations during specific times, vehicle type distribution, and the gender and age distribution of pedestrians.

[0125] Specific examples

[0126] For example, consider analyzing traffic data at a major intersection in a city.

[0127] HAPS captures the intersection for several hours and sends the video data to a server. The server then sends the video to an analysis center, where it is analyzed using generative artificial intelligence. The analysis center analyzes traffic volume, vehicle type, pedestrian gender and age, and sends the results back to the server. The server categorizes and saves the data and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time.

[0128] Prompt Sentence Examples

[0129] "Analyze traffic data for a specific area of ​​a specific city and generate a report with information on traffic volume, vehicle types, and pedestrian attributes. For example, I want to know the traffic conditions at major intersections from 8:00 to 9:00 AM."

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The server receives the video data from HAPS

[0133] Input: Real-time video data transmitted from an unmanned aerial vehicle (HAPS).

[0134] Specific operation: The server listens on a specified communication port (e.g., HTTP or FTP) and receives real-time video data from HAPS. The received data is stored in temporary storage.

[0135] Output: Video data saved in temporary storage.

[0136] Step 2:

[0137] The server converts the format of the video data

[0138] Input: Video data stored in temporary storage.

[0139] Specific operation: The server converts the format of the received video data into a format acceptable to the analysis center (e.g., converting from JPEG to MP4). This process uses an image conversion library (e.g., FFmpeg).

[0140] Output: Format converted video data.

[0141] Step 3:

[0142] The server sends the converted data to the analysis center.

[0143] Input: Format-converted video data.

[0144] How it works: The server uses an API call, such as an HTTP POST request, to send the converted video data to the analysis center.

[0145] Output: Video data sent to analysis center.

[0146] Step 4:

[0147] The analysis center analyzes the video data

[0148] Input: Video data sent to analysis center.

[0149] How it works: The analysis center uses generative artificial intelligence to analyze the received video data, including calculating traffic volume, classifying vehicle types, and estimating pedestrian attributes (age and gender).

[0150] Output: Analyzed traffic volume data, vehicle type data, and pedestrian attribute information data.

[0151] Step 5:

[0152] The server receives, classifies, and stores the analysis results

[0153] Input: Analysis result data sent from the analysis center.

[0154] How it works: The server receives data sent from the analysis center and classifies it into categories such as traffic volume data, vehicle data, pedestrian data, etc. The classified data is indexed for efficient search and stored in a database.

[0155] Output: Data categorized and stored in a database.

[0156] Step 6:

[0157] The server generates the report

[0158] Input: Classified data stored in a database.

[0159] Specific operation: The server reads a prepared report template based on the classified data, embeds the data in the template, and generates a report. The report is generated in PDF or HTML format, and graphs and tables are added to fit each format.

[0160] Output: Generated report in PDF and HTML format.

[0161] Step 7:

[0162] Accessing reports from your device

[0163] Input: The query information for the user to access (date, time, location, etc.).

[0164] How it works: A user accesses the web interface provided by the server using a device such as a PC or smartphone, selects a report for a specific date, time, or location, and the server sends a query to the database to retrieve the corresponding report.

[0165] Output: The report displayed on the terminal.

[0166] Step 8:

[0167] Users view and analyze reports

[0168] Input: Report data retrieved from the server.

[0169] What it does: The user views a report on their device, which includes graphs of traffic volume fluctuations, vehicle type distribution, and the gender and age distribution of pedestrians.

[0170] Output: The results and insights of the user's analysis.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] In modern cities, obtaining traffic information quickly and accurately and using that information to optimize route selection and operation management for autonomous vehicles is a critical issue. Conventional traffic information systems can only provide a limited range of information and lack real-time performance. Furthermore, manually analyzing and updating detailed traffic data requires a great deal of effort and time. There is a need to resolve these issues and build a system that provides highly accurate and rapid traffic information.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for generating and providing information for optimizing route selection and operation management of autonomous vehicles. This enables the efficient operation of autonomous vehicles by analyzing a wide range of traffic information obtained from the unmanned aerial vehicle in real time.

[0176] An "unmanned aerial vehicle" is an aircraft that is remotely controlled and operated using advanced technology, and is a device used to collect video data of a specific area.

[0177] "Video data" refers to digital data of real-time video of a specific area captured from the sky by an unmanned aerial vehicle.

[0178] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[0179] "Generative artificial intelligence" is an AI technology that uses machine learning techniques to find patterns in huge amounts of data and generate information that meets specific purposes.

[0180] "Traffic volume" is information indicating the number and density of vehicles traveling on a road during a specific time period.

[0181] "Vehicle type" is information indicating the type or classification of vehicles included in the traffic data.

[0182] "Pedestrian attribute information" is information that indicates characteristics of pedestrians, such as gender and age group, contained in the traffic data.

[0183] A "database" is an information management system that organizes and stores acquired information so that it can be retrieved as needed.

[0184] "Report" means an automatically generated summary of information based on analyzed and classified traffic data, in the form provided to the User.

[0185] A "user-friendly format" is a presentation of information that is designed to be easy for users to understand and operate.

[0186] "Information for optimizing route selection and operation management for autonomous vehicles" refers to information including traffic data and analysis results that enable autonomous vehicles to operate efficiently and safely.

[0187] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0188] composition

[0189] 1. Unmanned Aerial Vehicles (HAPS):

[0190] HAPS flies overhead and captures video data of a specific area in real time.

[0191] 2. Server:

[0192] The server has the function of receiving video data transmitted from HAPS.

[0193] The server sends the received video data to an analysis center.

[0194] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0195] The server generates and stores reports based on the classified data.

[0196] The server generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0197] 3. Analysis Center:

[0198] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0199] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0200] 4. Terminal:

[0201] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0202] The terminal displays operational management information for autonomous vehicles and is used for route selection.

[0203] operation

[0204] 1. Server receives video from HAPS:

[0205] The server receives real-time video data transmitted by the HAPS every hour. For example, it can acquire video data of a specific intersection or a wide area of ​​a city. The communication protocol is 5G or LoRaWAN.

[0206] 2. The server sends the data to the analysis center:

[0207] The server sends the received video data to an analysis center, which uses generative artificial intelligence (e.g., OpenAI's GPT-4®) to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0208] 3. The server receives, classifies, and saves the analysis results:

[0209] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) The classified data is stored in Amazon AWS (registered trademark) S3.

[0210] 4. Server generates reports and traffic control information:

[0211] Using the classified data, the server automatically generates reports detailing the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0212] In addition, it generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0213] 5. Displaying reports and traffic control information on the device:

[0214] Users can access the interface using their devices to view reports and traffic management information generated from the server's database. Users can select reports for specific dates, times, and locations, and use them to check traffic information and plan routes for autonomous vehicles.

[0215] Examples of concrete examples and prompts

[0216] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users access the system from their devices and can view detailed traffic data for that intersection for a specific date and time in a report, while also receiving suggestions for optimal routes for autonomous vehicles.

[0217] An example of a new prompt is:

[0218] "Analyze real-time video data sent from an unmanned aerial vehicle (HAPS) and extract information on traffic volume, vehicle types, and pedestrian attributes at specific intersections. Based on the results, generate a Python program that suggests an optimal route with minimal risk of delay."

[0219] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0221] Step 1:

[0222] The server receives video data from the HAPS. The input is real-time video data sent from the HAPS, and the output is video data stored in the server. The server receives this data using communication protocols such as 5G or LoRaWAN.

[0223] Step 2:

[0224] The server sends the received video data to the analysis center. The input is the video data stored on the server, and the output is the video data sent to the analysis center. The server transfers the data to the analysis center via the network.

[0225] Step 3:

[0226] The analysis center uses generative artificial intelligence (such as GPT-4) to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input is the video data sent to the analysis center, and the output is analyzed traffic information data. The generative artificial intelligence analyzes the video data and extracts specific patterns.

[0227] Step 4:

[0228] The server receives the analysis results from the analysis center. The input is the analysis result data sent from the analysis center, and the output is the analysis result data stored in the server. The server receives and stores the data from the analysis center.

[0229] Step 5:

[0230] The server categorizes the analyzed information and stores it in a database. The input is the analysis result data stored on the server, and the output is the categorized data stored in the database. The server organizes traffic volume, vehicle type, and pedestrian attribute information by category.

[0231] Step 6:

[0232] The server generates reports based on the classification data stored in the database. The input is the classification data in the database and the output is the generated report. The server automatically generates reports based on the classified data.

[0233] Step 7:

[0234] The server generates information to optimize route selection and operation management for autonomous vehicles. The input is the classification data in the database and the generated report, and the output is optimized operation information. The server proposes the optimal route taking into account real-time traffic information.

[0235] Step 8:

[0236] The terminal displays the reports and traffic control information sent from the server. The input is the reports and traffic control information sent from the server, and the output is the information displayed on the terminal screen. The user can check detailed traffic conditions and traffic information through the terminal interface.

[0237] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0238] This invention combines a system that analyzes traffic information based on video data from unmanned aerial vehicles (HAPS) and automatically generates user-friendly reports with an emotion engine that recognizes user emotions. This system receives video data from unmanned aerial vehicles, analyzes it at an analysis center, generates reports, and reflects user feedback through emotion recognition when providing the reports in a user-friendly format.

[0239] composition

[0240] 1. Unmanned aerial vehicle (HAPS)

[0241] HAPS flies overhead and captures video data of a specific area in real time.

[0242] 2. Server

[0243] The server has the function of receiving video data transmitted from HAPS.

[0244] The server sends the received video data to an analysis center.

[0245] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0246] The server generates and stores reports based on the classified data.

[0247] 3. Analysis Center

[0248] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0249] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0250] 4. Terminal

[0251] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0252] The device has a built-in emotion engine that recognizes the user's emotions and collects the user's emotion data.

[0253] operation

[0254] 1. The server receives the video from HAPS

[0255] The server receives and temporarily stores real-time video data transmitted by HAPS, which can capture images of the intersection or a wide area of ​​the city.

[0256] 2. The server sends the data to the analysis center

[0257] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0258] 3. The server receives, classifies, and saves the analysis results

[0259] After receiving the analysis result data from the analysis center, the server classifies the results into each data category and stores them in a database.

[0260] 4. The server generates the report

[0261] Using the classified data, the server automatically generates reports detailing traffic volume fluctuations during specific times of the day, vehicle distribution, pedestrian attributes, and more.

[0262] 5. Viewing reports on your device

[0263] By accessing the interface using a terminal, a user can view reports generated from the server's database. The terminal uses an emotion engine to collect emotional data from the user's facial expressions, voice, movements, etc., and adjusts the display of the report based on that data.

[0264] 6. User Emotion Recognition and Feedback

[0265] The emotion engine analyzes the user's emotional data, and if it determines that the content of a report is difficult to understand, for example, it automatically reflects the feedback, such as revising the content or providing additional explanations.

[0266] Specific examples

[0267] For example, to analyze traffic data at a major intersection in a city, HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server then receives the analysis results, classifies and saves them into categories, and automatically generates reports.

[0268] Users access the system from their devices and view detailed traffic data for a specific intersection for a specific date and time as a report. The device's emotion engine recognizes the user's emotions. For example, if the user is viewing the report with interest, the system will suggest related information or data for other time periods. Conversely, if the user looks confused, the system will respond by providing more detailed information in the report.

[0269] In this way, the present invention can provide highly accurate and wide-ranging traffic information in real time, which can significantly improve the efficiency of urban planning and traffic management. Furthermore, by recognizing users' emotions and reflecting their feedback, a more user-friendly system can be realized.

[0270] The processing flow will be explained below.

[0271] Step 1:

[0272] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0273] Step 2:

[0274] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0275] Step 3:

[0276] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0277] Step 4:

[0278] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[0279] Step 5:

[0280] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[0281] Step 6:

[0282] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[0283] Step 7:

[0284] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[0285] Step 8:

[0286] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[0287] Step 9:

[0288] The device's built-in emotion engine recognizes the user's emotions. When the user views a report, the emotion engine analyzes the user's facial expressions, voice, and movements to recognize the user's emotions in real time.

[0289] Step 10:

[0290] The device uses an emotion engine to analyze the user's emotional data. For example, if the user shows a confused expression, the device will display additional explanations. Conversely, if the user shows interest, the device will suggest related information or other reports.

[0291] Step 11:

[0292] The user reviews the provided reports and suggested information and obtains the information they need. They can then request further details or provide feedback.

[0293] In this way, the present invention realizes a system that recognizes the user's emotions and appropriately reflects feedback to provide reports and information that are easier to use and understand.

[0294] Example 2

[0295] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] Current traffic information analysis systems have limitations in efficiently analyzing wide-area video data acquired from unmanned aerial vehicles and providing it in a user-friendly format. Furthermore, they are unable to recognize user emotions in real time and incorporate feedback, limiting the extent to which they can improve usability. This creates problems that limit the efficiency of urban planning and traffic management.

[0297] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0298] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and storing it in a database, means for generating and storing reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for the terminal to recognize user emotions and collect emotion data, and means for adjusting the display of the report based on the emotion data. This makes it possible to efficiently and comprehensively analyze traffic information and provide it in a user-friendly format. Furthermore, by recognizing user emotions in real time and reflecting feedback based on that, an even easier-to-use system can be realized.

[0299] An "unmanned aerial vehicle" is an aircraft that flies at high altitudes for long periods of time and captures video data of a specific area.

[0300] "Video data" refers to data containing a wide range of visual information captured by an unmanned aerial vehicle.

[0301] "Server" refers to a device that receives, stores, transmits, and classifies, stores, and analyzes video data.

[0302] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[0303] "Generative artificial intelligence" is a technology that uses machine learning models to analyze data and generate new information.

[0304] "Traffic volume" is data that indicates the frequency and density of traffic observed at a specific time or location.

[0305] "Vehicle type" is information indicating the type of vehicle contained in the analyzed video data.

[0306] "Pedestrian attribute information" is data that indicates specific characteristics of a pedestrian, such as gender and age.

[0307] "Categorizing" is the process of organizing and storing analyzed information according to specific categories.

[0308] A "database" is a system for efficiently storing and managing large amounts of data.

[0309] A "report" is a document that summarizes specific information in an easy-to-understand manner, generated based on classified data.

[0310] A "user-friendly format" is a presentation format that provides information in a way that is easy for users to understand and use.

[0311] A "terminal" is a device that has an interface that allows a user to access the system and perform operations and checks.

[0312] "Emotion recognition" is the process of analyzing a user's emotions from their facial expressions and voice.

[0313] "Emotional data" is specific information about the analyzed user's emotions.

[0314] "Feedback" is the process by which a system adjusts the information and functionality it displays based on the user's emotional data.

[0315] This invention combines a system that analyzes traffic information based on video data from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports, with an emotion engine that recognizes the user's emotions. This system is implemented in the following form.

[0316] First, unmanned aerial vehicles (HAPS) fly over a specific area at high altitude for a long period of time, capturing wide-area video data in real time. This video data includes high-resolution video of 1080p. The unmanned aerial vehicles transmit this video data to a server in real time using RTSP (Real-Time Streaming Protocol). A specific example would be monitoring a major intersection or a city square.

[0317] The server receives this video data and temporarily stores it. The server is equipped with a high-performance network interface card (NIC) and large-capacity storage. The received video data is sent to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer, enabling secure and high-speed data transfer. An example of a specific prompt sent by the server is as follows:

[0318] Destination URL: https: / / analysis-center.example.com / analyze

[0319] Sending data: Video data batch

[0320] Sending method: POST

[0321] The analysis center uses computers equipped with high-performance GPUs to analyze video data using generative artificial intelligence (AI). Specifically, it uses Tensorflow's (registered trademark) object detection API and PyTorch to extract attribute information on traffic volume, vehicle types, and pedestrians in the video. This allows for detailed analysis of hourly traffic volume, the proportion of vehicle types, and the gender and age of pedestrians.

[0322] The server receives the analysis results sent from the analysis center and classifies and stores them in a database. MySQL (registered trademark) and PostgreSQL are commonly used relational databases for servers. By classifying and storing the analysis results by specific categories, the server can easily search and reference them later. Specific examples include "traffic volume data at intersection A" and "vehicle type data at intersection A."

[0323] Based on the classified data, the server automatically generates reports using Python data analysis and visualization libraries such as Pandas and Matplotlib, including line graphs showing the increase or decrease in traffic volume during specific times of day and pie charts showing the breakdown of vehicle types.

[0324] Users access reports using a terminal. The terminal uses front-end frameworks such as React and Angular to provide a user-friendly interface. When displaying reports, the terminal uses a built-in emotion engine to capture and analyze the user's facial expressions and voice in real time. This analysis is facilitated by the OpenCV library and IBM Watson (registered trademark) emotion analysis API. By analyzing the user's emotions, the system reflects the feedback in the report and adjusts the display content. For example, if the user shows a confused expression, the system automatically adds a supplementary explanation.

[0325] As described above, this invention can efficiently analyze a wide range of traffic information and provide user-friendly reports. Furthermore, by recognizing the user's emotions in real time and incorporating their feedback, it is possible to realize an even easier-to-use system.

[0326] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0327] Step 1:

[0328] The server receives video from the unmanned aerial vehicle (HAPS). The server uses the RTSP protocol to receive 1080p high-resolution video data transmitted from the unmanned aerial vehicle in real time and temporarily stores it. The input is the video data from the unmanned aerial vehicle, and the output is the temporarily stored video data. Specifically, the server makes full use of a network interface card (NIC) to receive data at high speed and with stability.

[0329] Step 2:

[0330] The server sends the data to the analysis center. The server then sends the temporarily stored video data to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer. The input is the temporarily stored video data, and the output is the video data sent to the analysis center. Specifically, the server batches the data and generates a prompt to send to the analysis center.

[0331] Step 3:

[0332] The analysis center analyzes the video data. The analysis center uses a computer equipped with a high-performance GPU to analyze the video data using a generative AI model. The input is the video data sent to the analysis center, and the output is analyzed traffic information, vehicle information, and pedestrian attribute information. Specifically, a machine learning model using TensorFlow and PyTorch detects vehicles and pedestrians in the video and analyzes their characteristics.

[0333] Step 4:

[0334] The server receives, classifies, and stores the analysis results. The server receives the analysis results sent from the analysis center and classifies and stores them in a relational database such as MySQL or PostgreSQL. The input is the analysis results from the analysis center, and the output is the analysis results that have been classified and stored in the database. Specifically, the server classifies the results into categories such as "traffic volume data," "vehicle type data," and "pedestrian attribute data."

[0335] Step 5:

[0336] The server generates the report. Based on the classified data, the server automatically generates the report using Python's Pandas and Matplotlib. The input is the classified data stored in the database, and the output is the generated report. Specifically, the server creates a line graph showing the increase or decrease in traffic volume during a specific time period, and a pie chart showing the proportion of vehicle types, and compiles these into a report.

[0337] Step 6:

[0338] The terminal displays the report. The terminal uses a front-end framework such as React or Angular to provide a user-friendly interface. The input is the generated report, and the output is the report displayed to the user. Specifically, the terminal updates the report display according to user operations.

[0339] Step 7:

[0340] The device recognizes the user's emotions and provides feedback. The device uses a built-in emotion engine to analyze the user's facial expressions and voice in real time. The input is the user's facial and voice data, and the output is analyzed emotion data. Specifically, the OpenCV library and IBM Watson emotion analysis API analyze the user's emotions and reflects the feedback in the report based on that. For example, if the user is confused, additional explanations will be displayed.

[0341] (Application example 2)

[0342] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0343] Conventional traffic information analysis systems can provide highly accurate traffic information in real time, but they lack feedback and display adjustments based on the user's emotional state. Furthermore, they lack flexible adjustments to avoid user stress and confusion, making it difficult to provide a user-friendly interface. Furthermore, it is necessary to recognize the emotions of drivers and passengers and provide optimal navigation and information display based on those emotions.

[0344] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving a report based on the categorized data, means for providing the generated report in a user-friendly format, means for recognizing the user's emotions using sensors in the vehicle, and means for adjusting the display or guidance based on the emotion recognition data. This enables the provision of real-time traffic information according to the user's emotions and safe and comfortable navigation.

[0345] An "unmanned aerial vehicle" is an aircraft that does not have a human pilot and that flies autonomously or by remote control.

[0346] "Video data" refers to data that includes visual information acquired using a camera, sensor, etc.

[0347] "Analysis Center" means a facility or system that processes and analyzes received data.

[0348] "Generative AI" is an AI that has the ability to learn large amounts of data and analyze and generate data that is given to it.

[0349] "Traffic volume" is an indicator that indicates the number of vehicles or people passing through a specific location within a certain period of time.

[0350] "Vehicle type" is a classification used to distinguish between different types of automobiles.

[0351] "Attribute information of pedestrians" is information including characteristics of pedestrians such as gender, age, and clothing.

[0352] A "database" is a system for efficiently storing, searching, and managing data.

[0353] "Report" means a report or summary of information generated based on analyzed data.

[0354] "User-friendly" is a term that describes something that is easy for users to use and understand.

[0355] A "sensor" is a device that detects a physical quantity (e.g., temperature, humidity, movement, etc.) and converts it into data.

[0356] "Emotion recognition data" is data used to identify emotions from human facial expressions, voice, movements, etc.

[0357] "Display adaptation" refers to changing how information is displayed in response to user factors.

[0358] "Guidance" is a term that means providing guidance to a destination or information.

[0359] The system that specifically embodies this invention is composed of an unmanned aerial vehicle (HAPS), a server, an analysis center, a database, and terminals and sensors within the autonomous vehicle.

[0360] The server first receives video data acquired from an unmanned aerial vehicle (HAPS). The unmanned aerial vehicle is capable of capturing images of a wide area and transmitting real-time video data. The received video data is temporarily stored by the server and then sent to the analysis center.

[0361] The analysis center uses generative AI to analyze the video data. Specifically, information on traffic volume, vehicle types, and pedestrian attributes is acquired. Examples of generative AI used include OpenAI's GPT and Google's DeepMind.

[0362] The analyzed data is sent to a server, which categorizes it and stores it in a database. The database can be MySQL or PostgreSQL. The server then automatically generates and stores reports based on the categorized data.

[0363] The terminals inside the autonomous vehicles are equipped with cameras and microphones, which allow them to recognize the emotions of the driver and passengers. Emotion recognition engines such as Microsoft® Azure® Cognitive Services and Amazon Rekognition are used for emotion recognition. Emotion data is sent to a server, which then adjusts the display and guidance based on this data.

[0364] As a concrete example, while an autonomous vehicle is driving in an urban area, HAPS captures real-time images of the surrounding traffic conditions. This image data is sent to a server and then analyzed as traffic information at an analysis center. The captured traffic information is then sent to the server and stored in a database. If the driver is feeling stressed, the sensor sends this emotional data to the server, which then generates a report suggesting the optimal detour route. The report is then displayed on the device, providing the driver with comfortable navigation.

[0365] An example prompt is:

[0366] "Explain the system that analyzes video data from HAPS, obtains traffic information, and provides the optimal navigation route based on the driver's emotional data."

[0367] This will enable the provision of real-time traffic information that responds to the user's emotions and safe and comfortable navigation.

[0368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0369] Step 1:

[0370] The server receives video data from the unmanned aerial vehicle. The HAPS captures a wide area and transmits real-time video data. This video data is input to the server and temporarily stored. The input data is stored in temporary storage and then goes to the next analysis step.

[0371] Step 2:

[0372] The server sends the received video data to the analysis center. Using a pre-configured communication protocol, the server transfers the stored video data to the analysis center. The input data is the video data from HAPS, and the output data is the video data transferred to the analysis center.

[0373] Step 3:

[0374] The analysis center uses generative artificial intelligence to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input data is the video data sent from HAPS, and a generative AI model (e.g., OpenAI's GPT or Google's DeepMind) is used. The analyzed data is extracted, and traffic information is obtained as output data. The obtained information includes the number of vehicles, vehicle type identification, and pedestrian attributes (gender, age, etc.).

[0375] Step 4:

[0376] The server receives traffic information sent from the analysis center, categorizes it into categories, and stores it in a database. The input data is the traffic information sent from the analysis center, and the output data is database entries organized by category. The server does this using a database management system (e.g., MySQL, PostgreSQL).

[0377] Step 5:

[0378] The server automatically generates and saves reports based on the classified data. The generated reports include information such as increases or decreases in traffic volume during specific time periods, vehicle distribution, and pedestrian attributes. The input data is the categorized traffic information stored in the database, and the output data is the generated report.

[0379] Step 6:

[0380] The terminal inside the autonomous vehicle displays the report and recognizes the user's emotions using sensors inside the vehicle. An emotion recognition engine (e.g., Microsoft Azure Cognitive Services, Amazon Rekognition) is used, and the input data is emotion information obtained from the camera and microphone. The output data is the analyzed emotion data.

[0381] Step 7:

[0382] The server adjusts the display and guidance based on emotion recognition data. The input data is emotion data sent from the emotion recognition engine, and the output data is adjusted navigation and information display. For example, if the driver is feeling stressed, the server will optimize the route and suggest a more comfortable route.

[0383] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0385] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0386] [Second embodiment]

[0387] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0388] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0390] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0394] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0397] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0398] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0399] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0400] composition

[0401] 1. Unmanned aerial vehicle (HAPS)

[0402] HAPS flies overhead and captures video data of a specific area in real time.

[0403] 2. Server

[0404] The server has the function of receiving video data transmitted from HAPS.

[0405] The server sends the received video data to an analysis center.

[0406] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0407] The server generates and stores reports based on the classified data.

[0408] 3. Analysis Center

[0409] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0410] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0411] 4. Terminal

[0412] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0413] operation

[0414] 1. The server receives the video from HAPS

[0415] Every hour, the server receives real-time video data transmitted by HAPS, allowing it to obtain video data for a specific intersection or a wide area of ​​a city.

[0416] 2. The server sends the data to the analysis center

[0417] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0418] 3. The server receives, classifies, and saves the analysis results

[0419] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) and stores the classified data in a database.

[0420] 4. The server generates the report

[0421] Using the classified data, the server automatically generates reports detailing, for example, the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0422] 5. Viewing reports on your device

[0423] Users can access the interface using their devices to view reports generated from the server's database. Users can select reports for specific dates, times, and locations, and check and analyze traffic information.

[0424] Specific examples

[0425] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection over several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time in the form of a report.

[0426] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0427] The processing flow will be explained below.

[0428] Step 1:

[0429] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0430] Step 2:

[0431] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0432] Step 3:

[0433] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0434] Step 4:

[0435] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[0436] Step 5:

[0437] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[0438] Step 6:

[0439] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[0440] Step 7:

[0441] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[0442] Step 8:

[0443] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[0444] Step 9:

[0445] The user can view the report they have selected. The user can check the details of the selected report and obtain the information they need. This allows the user to easily use traffic analysis data for a specific time period or area.

[0446] Example 1

[0447] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0448] Conventional traffic information collection systems have struggled to collect and analyze a wide range of data in real time and provide it in a user-friendly report format. Furthermore, classifying various data and generating reports requires a huge amount of time and effort, making them inadequate for urban planning and traffic management, which require rapid response. Furthermore, displaying analysis results makes it difficult to provide detailed visual data tailored to specific points in time and locations.

[0449] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0450] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, means for converting the format of the received video data, means for receiving analysis results from the analysis center and categorizing them into data categories, means for indexing the data stored in the database, means for generating reports based on templates, and means for accessing the reports from a terminal and displaying them at a specific date, time, or location. This enables the collection and analysis of wide-ranging and detailed traffic information in real time and the rapid provision of such information in a user-friendly format.

[0451] An "unmanned aerial vehicle" is an aircraft that flies under remote or automatic control and photographs and collects data over a specific area.

[0452] "Video data" refers to data containing visual information captured by unmanned aerial vehicles, capturing traffic conditions and people's movements.

[0453] An "analysis center" is a facility or system that analyzes received video data and extracts information such as traffic volume, vehicle types, and pedestrian attributes.

[0454] "Generative artificial intelligence" is an artificial intelligence model that has the ability to perform analysis of data and generate specific results.

[0455] "Classification by category" means dividing the acquired information based on specific attributes, such as traffic volume, vehicle type, and pedestrian attribute information.

[0456] A "database" is an information system for storing and managing classified data, enabling efficient searching and use of the data.

[0457] A "report" is a document or file that organizes analyzed data and presents it in a visually easy-to-understand format.

[0458] "User-friendly" means that the system or interface is designed and formatted to be easy for users to use.

[0459] "Format conversion" refers to the process of changing data into a format that is easy for the analysis center to process.

[0460] "Indexing" means adding identifiers or keys to data in a database to improve search efficiency.

[0461] A "template" is a predefined format or structure used to generate a report.

[0462] A "terminal" is a device used by a user to access the system and display data, examples of which include a PC or smartphone.

[0463] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle and automatically generates user-friendly reports. This system is configured as follows.

[0464] unmanned aerial vehicle

[0465] An unmanned aerial vehicle (HAPS) flies overhead and captures specific areas in real time. For example, a HAPS can capture images of major intersections in a city and send the video data to a server on the ground. The HAPS's flight altitude and camera resolution are set according to the situation, allowing it to capture detailed traffic conditions.

[0466] server

[0467] The server has the following functions:

[0468] 1. Video reception:

[0469] The server receives the video data sent from the HAPS via a monitoring port and stores it in temporary storage in real time. HTTP or FTP is used as the communication protocol.

[0470] 2. Data transmission:

[0471] The received video data is first converted into a format before being sent to the analysis center. For example, JPEG format images are converted into MP4 format videos and then sent to the analysis center.

[0472] 3. Receiving and classifying analysis results:

[0473] The analysis center returns the analysis results data, which includes information on traffic volume, vehicle types, and pedestrian attributes. The server categorizes this data and stores it in a database.

[0474] 4. Report Generation:

[0475] The server uses the classified data to automatically generate reports based on the specified template, which are saved in PDF or HTML format.

[0476] Analysis Center

[0477] The analysis center has the following functions:

[0478] 1. Data Analysis:

[0479] The received video data is analyzed using generative artificial intelligence (AI model), which calculates traffic volume, classifies vehicle types, and estimates the age and gender of pedestrians.

[0480] Terminal

[0481] Users access reports using a device (PC or smartphone).

[0482] 1. Access the interface:

[0483] Users access the server interface through a web browser or a dedicated app.

[0484] 2. Report Viewing:

[0485] Users can select reports for specific dates, times, and locations to view detailed traffic information, including, for example, traffic volume fluctuations during specific times, vehicle type distribution, and the gender and age distribution of pedestrians.

[0486] Specific examples

[0487] For example, consider analyzing traffic data at a major intersection in a city.

[0488] HAPS captures the intersection for several hours and sends the video data to a server. The server then sends the video to an analysis center, where it is analyzed using generative artificial intelligence. The analysis center analyzes traffic volume, vehicle type, pedestrian gender and age, and sends the results back to the server. The server categorizes and saves the data and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time.

[0489] Prompt Sentence Examples

[0490] "Analyze traffic data for a specific area of ​​a specific city and generate a report with information on traffic volume, vehicle types, and pedestrian attributes. For example, I want to know the traffic conditions at major intersections from 8:00 to 9:00 AM."

[0491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0492] Step 1:

[0493] The server receives the video data from HAPS

[0494] Input: Real-time video data transmitted from an unmanned aerial vehicle (HAPS).

[0495] Specific operation: The server listens on a specified communication port (e.g., HTTP or FTP) and receives real-time video data from HAPS. The received data is stored in temporary storage.

[0496] Output: Video data saved in temporary storage.

[0497] Step 2:

[0498] The server converts the format of the video data

[0499] Input: Video data stored in temporary storage.

[0500] Specific operation: The server converts the format of the received video data into a format acceptable to the analysis center (e.g., converting from JPEG to MP4). This process uses an image conversion library (e.g., FFmpeg).

[0501] Output: Format converted video data.

[0502] Step 3:

[0503] The server sends the converted data to the analysis center.

[0504] Input: Format-converted video data.

[0505] How it works: The server uses an API call, such as an HTTP POST request, to send the converted video data to the analysis center.

[0506] Output: Video data sent to analysis center.

[0507] Step 4:

[0508] The analysis center analyzes the video data

[0509] Input: Video data sent to analysis center.

[0510] How it works: The analysis center uses generative artificial intelligence to analyze the received video data, including calculating traffic volume, classifying vehicle types, and estimating pedestrian attributes (age and gender).

[0511] Output: Analyzed traffic volume data, vehicle type data, and pedestrian attribute information data.

[0512] Step 5:

[0513] The server receives, classifies, and stores the analysis results

[0514] Input: Analysis result data sent from the analysis center.

[0515] How it works: The server receives data sent from the analysis center and classifies it into categories such as traffic volume data, vehicle data, pedestrian data, etc. The classified data is indexed for efficient search and stored in a database.

[0516] Output: Data categorized and stored in a database.

[0517] Step 6:

[0518] The server generates the report

[0519] Input: Classified data stored in a database.

[0520] Specific operation: The server reads a prepared report template based on the classified data, embeds the data in the template, and generates a report. The report is generated in PDF or HTML format, and graphs and tables are added to fit each format.

[0521] Output: Generated report in PDF and HTML format.

[0522] Step 7:

[0523] Accessing reports from your device

[0524] Input: The query information for the user to access (date, time, location, etc.).

[0525] How it works: A user accesses the web interface provided by the server using a device such as a PC or smartphone, selects a report for a specific date, time, or location, and the server sends a query to the database to retrieve the corresponding report.

[0526] Output: The report displayed on the terminal.

[0527] Step 8:

[0528] Users view and analyze reports

[0529] Input: Report data retrieved from the server.

[0530] What it does: The user views a report on their device, which includes graphs of traffic volume fluctuations, vehicle type distribution, and the gender and age distribution of pedestrians.

[0531] Output: The results and insights of the user's analysis.

[0532] (Application example 1)

[0533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] In modern cities, obtaining traffic information quickly and accurately and using that information to optimize route selection and operation management for autonomous vehicles is a critical issue. Conventional traffic information systems can only provide a limited range of information and lack real-time performance. Furthermore, manually analyzing and updating detailed traffic data requires a great deal of effort and time. There is a need to resolve these issues and build a system that provides highly accurate and rapid traffic information.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0536] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for generating and providing information for optimizing route selection and operation management of autonomous vehicles. This enables the efficient operation of autonomous vehicles by analyzing a wide range of traffic information obtained from the unmanned aerial vehicle in real time.

[0537] An "unmanned aerial vehicle" is an aircraft that is remotely controlled and operated using advanced technology, and is a device used to collect video data of a specific area.

[0538] "Video data" refers to digital data of real-time video of a specific area captured from the sky by an unmanned aerial vehicle.

[0539] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[0540] "Generative artificial intelligence" is an AI technology that uses machine learning techniques to find patterns in huge amounts of data and generate information that meets specific purposes.

[0541] "Traffic volume" is information indicating the number and density of vehicles traveling on a road during a specific time period.

[0542] "Vehicle type" is information indicating the type or classification of vehicles included in the traffic data.

[0543] "Pedestrian attribute information" is information that indicates characteristics of pedestrians, such as gender and age group, contained in the traffic data.

[0544] A "database" is an information management system that organizes and stores acquired information so that it can be retrieved as needed.

[0545] "Report" means an automatically generated summary of information based on analyzed and classified traffic data, in the form provided to the User.

[0546] A "user-friendly format" is a presentation of information that is designed to be easy for users to understand and operate.

[0547] "Information for optimizing route selection and operation management for autonomous vehicles" refers to information including traffic data and analysis results that enable autonomous vehicles to operate efficiently and safely.

[0548] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0549] composition

[0550] 1. Unmanned Aerial Vehicles (HAPS):

[0551] HAPS flies overhead and captures video data of a specific area in real time.

[0552] 2. Server:

[0553] The server has the function of receiving video data transmitted from HAPS.

[0554] The server sends the received video data to an analysis center.

[0555] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0556] The server generates and stores reports based on the classified data.

[0557] The server generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0558] 3. Analysis Center:

[0559] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0560] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0561] 4. Terminal:

[0562] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0563] The terminal displays operational management information for autonomous vehicles and is used for route selection.

[0564] operation

[0565] 1. Server receives video from HAPS:

[0566] The server receives real-time video data transmitted by the HAPS every hour. For example, it can acquire video data of a specific intersection or a wide area of ​​a city. The communication protocol is 5G or LoRaWAN.

[0567] 2. The server sends the data to the analysis center:

[0568] The server sends the received video data to an analysis center, where it uses generative artificial intelligence (e.g., OpenAI's GPT-4) to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0569] 3. The server receives, classifies, and saves the analysis results:

[0570] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) The classified data is stored in Amazon AWS S3.

[0571] 4. Server generates reports and traffic control information:

[0572] Using the classified data, the server automatically generates reports detailing the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0573] In addition, it generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0574] 5. Displaying reports and traffic control information on the device:

[0575] Users can access the interface using their devices to view reports and traffic management information generated from the server's database. Users can select reports for specific dates, times, and locations, and use them to check traffic information and plan routes for autonomous vehicles.

[0576] Examples of concrete examples and prompts

[0577] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users access the system from their devices and can view detailed traffic data for that intersection for a specific date and time in a report, while also receiving suggestions for optimal routes for autonomous vehicles.

[0578] An example of a new prompt is:

[0579] "Analyze real-time video data sent from an unmanned aerial vehicle (HAPS) and extract information on traffic volume, vehicle types, and pedestrian attributes at specific intersections. Based on the results, generate a Python program that suggests an optimal route with minimal risk of delay."

[0580] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0581] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0582] Step 1:

[0583] The server receives video data from the HAPS. The input is real-time video data sent from the HAPS, and the output is video data stored in the server. The server receives this data using communication protocols such as 5G or LoRaWAN.

[0584] Step 2:

[0585] The server sends the received video data to the analysis center. The input is the video data stored on the server, and the output is the video data sent to the analysis center. The server transfers the data to the analysis center via the network.

[0586] Step 3:

[0587] The analysis center uses generative artificial intelligence (such as GPT-4) to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input is the video data sent to the analysis center, and the output is analyzed traffic information data. The generative artificial intelligence analyzes the video data and extracts specific patterns.

[0588] Step 4:

[0589] The server receives the analysis results from the analysis center. The input is the analysis result data sent from the analysis center, and the output is the analysis result data stored in the server. The server receives and stores the data from the analysis center.

[0590] Step 5:

[0591] The server categorizes the analyzed information and stores it in a database. The input is the analysis result data stored on the server, and the output is the categorized data stored in the database. The server organizes traffic volume, vehicle type, and pedestrian attribute information by category.

[0592] Step 6:

[0593] The server generates reports based on the classification data stored in the database. The input is the classification data in the database and the output is the generated report. The server automatically generates reports based on the classified data.

[0594] Step 7:

[0595] The server generates information to optimize route selection and operation management for autonomous vehicles. The input is the classification data in the database and the generated report, and the output is optimized operation information. The server proposes the optimal route taking into account real-time traffic information.

[0596] Step 8:

[0597] The terminal displays the reports and traffic control information sent from the server. The input is the reports and traffic control information sent from the server, and the output is the information displayed on the terminal screen. The user can check detailed traffic conditions and traffic information through the terminal interface.

[0598] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0599] This invention combines a system that analyzes traffic information based on video data from unmanned aerial vehicles (HAPS) and automatically generates user-friendly reports with an emotion engine that recognizes user emotions. This system receives video data from unmanned aerial vehicles, analyzes it at an analysis center, generates reports, and reflects user feedback through emotion recognition when providing the reports in a user-friendly format.

[0600] composition

[0601] 1. Unmanned aerial vehicle (HAPS)

[0602] HAPS flies overhead and captures video data of a specific area in real time.

[0603] 2. Server

[0604] The server has the function of receiving video data transmitted from HAPS.

[0605] The server sends the received video data to an analysis center.

[0606] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0607] The server generates and stores reports based on the classified data.

[0608] 3. Analysis Center

[0609] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0610] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0611] 4. Terminal

[0612] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0613] The device has a built-in emotion engine that recognizes the user's emotions and collects the user's emotion data.

[0614] operation

[0615] 1. The server receives the video from HAPS

[0616] The server receives and temporarily stores real-time video data transmitted by HAPS, which can capture images of the intersection or a wide area of ​​the city.

[0617] 2. The server sends the data to the analysis center

[0618] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0619] 3. The server receives, classifies, and saves the analysis results

[0620] After receiving the analysis result data from the analysis center, the server classifies the results into each data category and stores them in a database.

[0621] 4. The server generates the report

[0622] Using the classified data, the server automatically generates reports detailing traffic volume fluctuations during specific times of the day, vehicle distribution, pedestrian attributes, and more.

[0623] 5. Viewing reports on your device

[0624] By accessing the interface using a terminal, a user can view reports generated from the server's database. The terminal uses an emotion engine to collect emotional data from the user's facial expressions, voice, movements, etc., and adjusts the display of the report based on that data.

[0625] 6. User Emotion Recognition and Feedback

[0626] The emotion engine analyzes the user's emotional data, and if it determines that the content of a report is difficult to understand, for example, it automatically reflects the feedback, such as revising the content or providing additional explanations.

[0627] Specific examples

[0628] For example, to analyze traffic data at a major intersection in a city, HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server then receives the analysis results, classifies and saves them into categories, and automatically generates reports.

[0629] Users access the system from their devices and view detailed traffic data for a specific intersection for a specific date and time as a report. The device's emotion engine recognizes the user's emotions. For example, if the user is viewing the report with interest, the system will suggest related information or data for other time periods. Conversely, if the user looks confused, the system will respond by providing more detailed information in the report.

[0630] In this way, the present invention can provide highly accurate and wide-ranging traffic information in real time, which can significantly improve the efficiency of urban planning and traffic management. Furthermore, by recognizing users' emotions and reflecting their feedback, a more user-friendly system can be realized.

[0631] The processing flow will be explained below.

[0632] Step 1:

[0633] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0634] Step 2:

[0635] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0636] Step 3:

[0637] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0638] Step 4:

[0639] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[0640] Step 5:

[0641] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[0642] Step 6:

[0643] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[0644] Step 7:

[0645] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[0646] Step 8:

[0647] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[0648] Step 9:

[0649] The device's built-in emotion engine recognizes the user's emotions. When the user views a report, the emotion engine analyzes the user's facial expressions, voice, and movements to recognize the user's emotions in real time.

[0650] Step 10:

[0651] The device uses an emotion engine to analyze the user's emotional data. For example, if the user shows a confused expression, the device will display additional explanations. Conversely, if the user shows interest, the device will suggest related information or other reports.

[0652] Step 11:

[0653] The user reviews the provided reports and suggested information and obtains the information they need. They can then request further details or provide feedback.

[0654] In this way, the present invention realizes a system that recognizes the user's emotions and appropriately reflects feedback to provide reports and information that are easier to use and understand.

[0655] Example 2

[0656] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Current traffic information analysis systems have limitations in efficiently analyzing wide-area video data acquired from unmanned aerial vehicles and providing it in a user-friendly format. Furthermore, they are unable to recognize user emotions in real time and incorporate feedback, limiting the extent to which they can improve usability. This creates problems that limit the efficiency of urban planning and traffic management.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0659] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and storing it in a database, means for generating and storing reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for the terminal to recognize user emotions and collect emotion data, and means for adjusting the display of the report based on the emotion data. This makes it possible to efficiently and comprehensively analyze traffic information and provide it in a user-friendly format. Furthermore, by recognizing user emotions in real time and reflecting feedback based on that, an even easier-to-use system can be realized.

[0660] An "unmanned aerial vehicle" is an aircraft that flies at high altitudes for long periods of time and captures video data of a specific area.

[0661] "Video data" refers to data containing a wide range of visual information captured by an unmanned aerial vehicle.

[0662] "Server" refers to a device that receives, stores, transmits, and classifies, stores, and analyzes video data.

[0663] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[0664] "Generative artificial intelligence" is a technology that uses machine learning models to analyze data and generate new information.

[0665] "Traffic volume" is data that indicates the frequency and density of traffic observed at a specific time or location.

[0666] "Vehicle type" is information indicating the type of vehicle contained in the analyzed video data.

[0667] "Pedestrian attribute information" is data that indicates specific characteristics of a pedestrian, such as gender and age.

[0668] "Categorizing" is the process of organizing and storing analyzed information according to specific categories.

[0669] A "database" is a system for efficiently storing and managing large amounts of data.

[0670] A "report" is a document that summarizes specific information in an easy-to-understand manner, generated based on classified data.

[0671] A "user-friendly format" is a presentation format that provides information in a way that is easy for users to understand and use.

[0672] A "terminal" is a device that has an interface that allows a user to access the system and perform operations and checks.

[0673] "Emotion recognition" is the process of analyzing a user's emotions from their facial expressions and voice.

[0674] "Emotional data" is specific information about the analyzed user's emotions.

[0675] "Feedback" is the process by which a system adjusts the information and functionality it displays based on the user's emotional data.

[0676] This invention combines a system that analyzes traffic information based on video data from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports, with an emotion engine that recognizes the user's emotions. This system is implemented in the following form.

[0677] First, unmanned aerial vehicles (HAPS) fly over a specific area at high altitude for a long period of time, capturing wide-area video data in real time. This video data includes high-resolution video of 1080p. The unmanned aerial vehicles transmit this video data to a server in real time using RTSP (Real-Time Streaming Protocol). A specific example would be monitoring a major intersection or a city square.

[0678] The server receives this video data and temporarily stores it. The server is equipped with a high-performance network interface card (NIC) and large-capacity storage. The received video data is sent to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer, enabling secure and high-speed data transfer. An example of a specific prompt sent by the server is as follows:

[0679] Destination URL: https: / / analysis-center.example.com / analyze

[0680] Sending data: Video data batch

[0681] Sending method: POST

[0682] The analysis center uses computers equipped with high-performance GPUs to analyze video data using generative artificial intelligence (AI). Specifically, TensorFlow's object detection API and PyTorch are used to extract attribute information on traffic volume, vehicle types, and pedestrians in the video. This allows for detailed analysis of hourly traffic volume, the proportion of vehicle types, and the gender and age of pedestrians.

[0683] The server receives the analysis results sent from the analysis center and classifies and stores them in a database. MySQL or PostgreSQL are commonly used relational databases for servers. By classifying and storing the analysis results by specific categories, the server can easily search and reference them later. Specific examples include "traffic volume data at intersection A" and "vehicle type data at intersection A."

[0684] Based on the classified data, the server automatically generates reports using Python data analysis and visualization libraries such as Pandas and Matplotlib, including line graphs showing the increase or decrease in traffic volume during specific times of day and pie charts showing the breakdown of vehicle types.

[0685] Users access reports using a terminal. Front-end frameworks such as React and Angular are used on the terminal to provide a user-friendly interface. When displaying reports, the terminal uses a built-in emotion engine to capture and analyze the user's facial expressions and voice in real time. This analysis is facilitated by the OpenCV library and IBM Watson's emotion analysis API. By analyzing the user's emotions, the system reflects the feedback in the report and adjusts the display content. For example, if the user shows a confused expression, the system automatically adds a supplementary explanation.

[0686] As described above, this invention can efficiently analyze a wide range of traffic information and provide user-friendly reports. Furthermore, by recognizing the user's emotions in real time and incorporating their feedback, it is possible to realize an even easier-to-use system.

[0687] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0688] Step 1:

[0689] The server receives video from the unmanned aerial vehicle (HAPS). The server uses the RTSP protocol to receive 1080p high-resolution video data transmitted from the unmanned aerial vehicle in real time and temporarily stores it. The input is the video data from the unmanned aerial vehicle, and the output is the temporarily stored video data. Specifically, the server makes full use of a network interface card (NIC) to receive data at high speed and with stability.

[0690] Step 2:

[0691] The server sends the data to the analysis center. The server then sends the temporarily stored video data to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer. The input is the temporarily stored video data, and the output is the video data sent to the analysis center. Specifically, the server batches the data and generates a prompt to send to the analysis center.

[0692] Step 3:

[0693] The analysis center analyzes the video data. The analysis center uses a computer equipped with a high-performance GPU to analyze the video data using a generative AI model. The input is the video data sent to the analysis center, and the output is analyzed traffic information, vehicle information, and pedestrian attribute information. Specifically, a machine learning model using TensorFlow and PyTorch detects vehicles and pedestrians in the video and analyzes their characteristics.

[0694] Step 4:

[0695] The server receives, classifies, and stores the analysis results. The server receives the analysis results sent from the analysis center and classifies and stores them in a relational database such as MySQL or PostgreSQL. The input is the analysis results from the analysis center, and the output is the analysis results that have been classified and stored in the database. Specifically, the server classifies the results into categories such as "traffic volume data," "vehicle type data," and "pedestrian attribute data."

[0696] Step 5:

[0697] The server generates the report. Based on the classified data, the server automatically generates the report using Python's Pandas and Matplotlib. The input is the classified data stored in the database, and the output is the generated report. Specifically, the server creates a line graph showing the increase or decrease in traffic volume during a specific time period, and a pie chart showing the proportion of vehicle types, and compiles these into a report.

[0698] Step 6:

[0699] The terminal displays the report. The terminal uses a front-end framework such as React or Angular to provide a user-friendly interface. The input is the generated report, and the output is the report displayed to the user. Specifically, the terminal updates the report display according to user operations.

[0700] Step 7:

[0701] The device recognizes the user's emotions and provides feedback. The device uses a built-in emotion engine to analyze the user's facial expressions and voice in real time. The input is the user's facial and voice data, and the output is analyzed emotion data. Specifically, the OpenCV library and IBM Watson emotion analysis API analyze the user's emotions and reflects the feedback in the report based on that. For example, if the user is confused, additional explanations will be displayed.

[0702] (Application example 2)

[0703] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0704] Conventional traffic information analysis systems can provide highly accurate traffic information in real time, but they lack feedback and display adjustments based on the user's emotional state. Furthermore, they lack flexible adjustments to avoid user stress and confusion, making it difficult to provide a user-friendly interface. Furthermore, it is necessary to recognize the emotions of drivers and passengers and provide optimal navigation and information display based on those emotions.

[0705] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving a report based on the categorized data, means for providing the generated report in a user-friendly format, means for recognizing the user's emotions using sensors in the vehicle, and means for adjusting the display or guidance based on the emotion recognition data. This enables the provision of real-time traffic information according to the user's emotions and safe and comfortable navigation.

[0706] An "unmanned aerial vehicle" is an aircraft that does not have a human pilot and that flies autonomously or by remote control.

[0707] "Video data" refers to data that includes visual information acquired using a camera, sensor, etc.

[0708] "Analysis Center" means a facility or system that processes and analyzes received data.

[0709] "Generative AI" is an AI that has the ability to learn large amounts of data and analyze and generate data that is given to it.

[0710] "Traffic volume" is an indicator that indicates the number of vehicles or people passing through a specific location within a certain period of time.

[0711] "Vehicle type" is a classification used to distinguish between different types of automobiles.

[0712] "Attribute information of pedestrians" is information including characteristics of pedestrians such as gender, age, and clothing.

[0713] A "database" is a system for efficiently storing, searching, and managing data.

[0714] "Report" means a report or summary of information generated based on analyzed data.

[0715] "User-friendly" is a term that describes something that is easy for users to use and understand.

[0716] A "sensor" is a device that detects a physical quantity (e.g., temperature, humidity, movement, etc.) and converts it into data.

[0717] "Emotion recognition data" is data used to identify emotions from human facial expressions, voice, movements, etc.

[0718] "Display adaptation" refers to changing how information is displayed in response to user factors.

[0719] "Guidance" is a term that means providing guidance to a destination or information.

[0720] The system that specifically embodies this invention is composed of an unmanned aerial vehicle (HAPS), a server, an analysis center, a database, and terminals and sensors within the autonomous vehicle.

[0721] The server first receives video data acquired from an unmanned aerial vehicle (HAPS). The unmanned aerial vehicle is capable of capturing images of a wide area and transmitting real-time video data. The received video data is temporarily stored by the server and then sent to the analysis center.

[0722] The analysis center uses generative artificial intelligence to analyze the video data. Specifically, information on traffic volume, vehicle types, and pedestrian attributes is obtained. Examples of generative artificial intelligence used include OpenAI's GPT and Google's DeepMind.

[0723] The analyzed data is sent to a server, which categorizes it and stores it in a database. The database can be MySQL or PostgreSQL. The server then automatically generates and stores reports based on the categorized data.

[0724] The devices inside the autonomous vehicles are equipped with cameras and microphones that detect the emotions of the driver and passengers. Emotion recognition engines such as Microsoft Azure Cognitive Services and Amazon Rekognition are used for emotion recognition. Emotion data is sent to a server, which then adjusts the display and guidance based on this data.

[0725] As a concrete example, while an autonomous vehicle is driving in an urban area, HAPS captures real-time images of the surrounding traffic conditions. This image data is sent to a server and then analyzed as traffic information at an analysis center. The captured traffic information is then sent to the server and stored in a database. If the driver is feeling stressed, the sensor sends this emotional data to the server, which then generates a report suggesting the optimal detour route. The report is then displayed on the device, providing the driver with comfortable navigation.

[0726] An example prompt is:

[0727] "Explain the system that analyzes video data from HAPS, obtains traffic information, and provides the optimal navigation route based on the driver's emotional data."

[0728] This will enable the provision of real-time traffic information that responds to the user's emotions and safe and comfortable navigation.

[0729] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0730] Step 1:

[0731] The server receives video data from the unmanned aerial vehicle. The HAPS captures a wide area and transmits real-time video data. This video data is input to the server and temporarily stored. The input data is stored in temporary storage and then goes to the next analysis step.

[0732] Step 2:

[0733] The server sends the received video data to the analysis center. Using a pre-configured communication protocol, the server transfers the stored video data to the analysis center. The input data is the video data from HAPS, and the output data is the video data transferred to the analysis center.

[0734] Step 3:

[0735] The analysis center uses generative artificial intelligence to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input data is the video data sent from HAPS, and a generative AI model (e.g., OpenAI's GPT or Google's DeepMind) is used. The analyzed data is extracted, and traffic information is obtained as output data. The obtained information includes the number of vehicles, vehicle type identification, and pedestrian attributes (gender, age, etc.).

[0736] Step 4:

[0737] The server receives traffic information sent from the analysis center, categorizes it into categories, and stores it in a database. The input data is the traffic information sent from the analysis center, and the output data is database entries organized by category. The server does this using a database management system (e.g., MySQL, PostgreSQL).

[0738] Step 5:

[0739] The server automatically generates and saves reports based on the classified data. The generated reports include information such as increases or decreases in traffic volume during specific time periods, vehicle distribution, and pedestrian attributes. The input data is the categorized traffic information stored in the database, and the output data is the generated report.

[0740] Step 6:

[0741] The terminal inside the autonomous vehicle displays the report and recognizes the user's emotions using sensors inside the vehicle. An emotion recognition engine (e.g., Microsoft Azure Cognitive Services, Amazon Rekognition) is used, and the input data is emotion information obtained from the camera and microphone. The output data is the analyzed emotion data.

[0742] Step 7:

[0743] The server adjusts the display and guidance based on emotion recognition data. The input data is emotion data sent from the emotion recognition engine, and the output data is adjusted navigation and information display. For example, if the driver is feeling stressed, the server will optimize the route and suggest a more comfortable route.

[0744] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0745] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0746] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0747] [Third embodiment]

[0748] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0749] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0750] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0751] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0752] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0753] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0754] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0755] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0756] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0757] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0758] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0759] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0760] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0761] composition

[0762] 1. Unmanned aerial vehicle (HAPS)

[0763] HAPS flies overhead and captures video data of a specific area in real time.

[0764] 2. Server

[0765] The server has the function of receiving video data transmitted from HAPS.

[0766] The server sends the received video data to an analysis center.

[0767] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0768] The server generates and stores reports based on the classified data.

[0769] 3. Analysis Center

[0770] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0771] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0772] 4. Terminal

[0773] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0774] operation

[0775] 1. The server receives the video from HAPS

[0776] Every hour, the server receives real-time video data transmitted by HAPS, allowing it to obtain video data for a specific intersection or a wide area of ​​a city.

[0777] 2. The server sends the data to the analysis center

[0778] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0779] 3. The server receives, classifies, and saves the analysis results

[0780] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) and stores the classified data in a database.

[0781] 4. The server generates the report

[0782] Using the classified data, the server automatically generates reports detailing, for example, the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0783] 5. Viewing reports on your device

[0784] Users can access the interface using their devices to view reports generated from the server's database. Users can select reports for specific dates, times, and locations, and check and analyze traffic information.

[0785] Specific examples

[0786] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection over several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time in the form of a report.

[0787] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0788] The processing flow will be explained below.

[0789] Step 1:

[0790] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0791] Step 2:

[0792] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0793] Step 3:

[0794] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0795] Step 4:

[0796] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[0797] Step 5:

[0798] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[0799] Step 6:

[0800] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[0801] Step 7:

[0802] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[0803] Step 8:

[0804] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[0805] Step 9:

[0806] The user can view the report they have selected. The user can check the details of the selected report and obtain the information they need. This allows the user to easily use traffic analysis data for a specific time period or area.

[0807] Example 1

[0808] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0809] Conventional traffic information collection systems have struggled to collect and analyze a wide range of data in real time and provide it in a user-friendly report format. Furthermore, classifying various data and generating reports requires a huge amount of time and effort, making them inadequate for urban planning and traffic management, which require rapid response. Furthermore, displaying analysis results makes it difficult to provide detailed visual data tailored to specific points in time and locations.

[0810] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0811] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, means for converting the format of the received video data, means for receiving analysis results from the analysis center and categorizing them into data categories, means for indexing the data stored in the database, means for generating reports based on templates, and means for accessing the reports from a terminal and displaying them at a specific date, time, or location. This enables the collection and analysis of wide-ranging and detailed traffic information in real time and the rapid provision of such information in a user-friendly format.

[0812] An "unmanned aerial vehicle" is an aircraft that flies under remote or automatic control and photographs and collects data over a specific area.

[0813] "Video data" refers to data containing visual information captured by unmanned aerial vehicles, capturing traffic conditions and people's movements.

[0814] An "analysis center" is a facility or system that analyzes received video data and extracts information such as traffic volume, vehicle types, and pedestrian attributes.

[0815] "Generative artificial intelligence" is an artificial intelligence model that has the ability to perform analysis of data and generate specific results.

[0816] "Classification by category" means dividing the acquired information based on specific attributes, such as traffic volume, vehicle type, and pedestrian attribute information.

[0817] A "database" is an information system for storing and managing classified data, enabling efficient searching and use of the data.

[0818] A "report" is a document or file that organizes analyzed data and presents it in a visually easy-to-understand format.

[0819] "User-friendly" means that the system or interface is designed and formatted to be easy for users to use.

[0820] "Format conversion" refers to the process of changing data into a format that is easy for the analysis center to process.

[0821] "Indexing" means adding identifiers or keys to data in a database to improve search efficiency.

[0822] A "template" is a predefined format or structure used to generate a report.

[0823] A "terminal" is a device used by a user to access the system and display data, examples of which include a PC or smartphone.

[0824] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle and automatically generates user-friendly reports. This system is configured as follows.

[0825] unmanned aerial vehicle

[0826] An unmanned aerial vehicle (HAPS) flies overhead and captures specific areas in real time. For example, a HAPS can capture images of major intersections in a city and send the video data to a server on the ground. The HAPS's flight altitude and camera resolution are set according to the situation, allowing it to capture detailed traffic conditions.

[0827] server

[0828] The server has the following functions:

[0829] 1. Video reception:

[0830] The server receives the video data sent from the HAPS via a monitoring port and stores it in temporary storage in real time. HTTP or FTP is used as the communication protocol.

[0831] 2. Data transmission:

[0832] The received video data is first converted into a format before being sent to the analysis center. For example, JPEG format images are converted into MP4 format videos and then sent to the analysis center.

[0833] 3. Receiving and classifying analysis results:

[0834] The analysis center returns the analysis results data, which includes information on traffic volume, vehicle types, and pedestrian attributes. The server categorizes this data and stores it in a database.

[0835] 4. Report Generation:

[0836] The server uses the classified data to automatically generate reports based on the specified template, which are saved in PDF or HTML format.

[0837] Analysis Center

[0838] The analysis center has the following functions:

[0839] 1. Data Analysis:

[0840] The received video data is analyzed using generative artificial intelligence (AI model), which calculates traffic volume, classifies vehicle types, and estimates the age and gender of pedestrians.

[0841] Terminal

[0842] Users access reports using a device (PC or smartphone).

[0843] 1. Access the interface:

[0844] Users access the server interface through a web browser or a dedicated app.

[0845] 2. Report Viewing:

[0846] Users can select reports for specific dates, times, and locations to view detailed traffic information, including, for example, traffic volume fluctuations during specific times, vehicle type distribution, and the gender and age distribution of pedestrians.

[0847] Specific examples

[0848] For example, consider analyzing traffic data at a major intersection in a city.

[0849] HAPS captures the intersection for several hours and sends the video data to a server. The server then sends the video to an analysis center, where it is analyzed using generative artificial intelligence. The analysis center analyzes traffic volume, vehicle type, pedestrian gender and age, and sends the results back to the server. The server categorizes and saves the data and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time.

[0850] Prompt Sentence Examples

[0851] "Analyze traffic data for a specific area of ​​a specific city and generate a report with information on traffic volume, vehicle types, and pedestrian attributes. For example, I want to know the traffic conditions at major intersections from 8:00 to 9:00 AM."

[0852] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0853] Step 1:

[0854] The server receives the video data from HAPS

[0855] Input: Real-time video data transmitted from an unmanned aerial vehicle (HAPS).

[0856] Specific operation: The server listens on a specified communication port (e.g., HTTP or FTP) and receives real-time video data from HAPS. The received data is stored in temporary storage.

[0857] Output: Video data saved in temporary storage.

[0858] Step 2:

[0859] The server converts the format of the video data

[0860] Input: Video data stored in temporary storage.

[0861] Specific operation: The server converts the format of the received video data into a format acceptable to the analysis center (e.g., converting from JPEG to MP4). This process uses an image conversion library (e.g., FFmpeg).

[0862] Output: Format converted video data.

[0863] Step 3:

[0864] The server sends the converted data to the analysis center.

[0865] Input: Format-converted video data.

[0866] How it works: The server uses an API call, such as an HTTP POST request, to send the converted video data to the analysis center.

[0867] Output: Video data sent to analysis center.

[0868] Step 4:

[0869] The analysis center analyzes the video data

[0870] Input: Video data sent to analysis center.

[0871] How it works: The analysis center uses generative artificial intelligence to analyze the received video data, including calculating traffic volume, classifying vehicle types, and estimating pedestrian attributes (age and gender).

[0872] Output: Analyzed traffic volume data, vehicle type data, and pedestrian attribute information data.

[0873] Step 5:

[0874] The server receives, classifies, and stores the analysis results

[0875] Input: Analysis result data sent from the analysis center.

[0876] How it works: The server receives data sent from the analysis center and classifies it into categories such as traffic volume data, vehicle data, pedestrian data, etc. The classified data is indexed for efficient search and stored in a database.

[0877] Output: Data categorized and stored in a database.

[0878] Step 6:

[0879] The server generates the report

[0880] Input: Classified data stored in a database.

[0881] Specific operation: The server reads a prepared report template based on the classified data, embeds the data in the template, and generates a report. The report is generated in PDF or HTML format, and graphs and tables are added to fit each format.

[0882] Output: Generated report in PDF and HTML format.

[0883] Step 7:

[0884] Accessing reports from your device

[0885] Input: The query information for the user to access (date, time, location, etc.).

[0886] How it works: A user accesses the web interface provided by the server using a device such as a PC or smartphone, selects a report for a specific date, time, or location, and the server sends a query to the database to retrieve the corresponding report.

[0887] Output: The report displayed on the terminal.

[0888] Step 8:

[0889] Users view and analyze reports

[0890] Input: Report data retrieved from the server.

[0891] What it does: The user views a report on their device, which includes graphs of traffic volume fluctuations, vehicle type distribution, and the gender and age distribution of pedestrians.

[0892] Output: The results and insights of the user's analysis.

[0893] (Application example 1)

[0894] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] In modern cities, obtaining traffic information quickly and accurately and using that information to optimize route selection and operation management for autonomous vehicles is a critical issue. Conventional traffic information systems can only provide a limited range of information and lack real-time performance. Furthermore, manually analyzing and updating detailed traffic data requires a great deal of effort and time. There is a need to resolve these issues and build a system that provides highly accurate and rapid traffic information.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0897] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for generating and providing information for optimizing route selection and operation management of autonomous vehicles. This enables the efficient operation of autonomous vehicles by analyzing a wide range of traffic information obtained from the unmanned aerial vehicle in real time.

[0898] An "unmanned aerial vehicle" is an aircraft that is remotely controlled and operated using advanced technology, and is a device used to collect video data of a specific area.

[0899] "Video data" refers to digital data of real-time video of a specific area captured from the sky by an unmanned aerial vehicle.

[0900] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[0901] "Generative artificial intelligence" is an AI technology that uses machine learning techniques to find patterns in huge amounts of data and generate information that meets specific purposes.

[0902] "Traffic volume" is information indicating the number and density of vehicles traveling on a road during a specific time period.

[0903] "Vehicle type" is information indicating the type or classification of vehicles included in the traffic data.

[0904] "Pedestrian attribute information" is information that indicates characteristics of pedestrians, such as gender and age group, contained in the traffic data.

[0905] A "database" is an information management system that organizes and stores acquired information so that it can be retrieved as needed.

[0906] "Report" means an automatically generated summary of information based on analyzed and classified traffic data, in the form provided to the User.

[0907] A "user-friendly format" is a presentation of information that is designed to be easy for users to understand and operate.

[0908] "Information for optimizing route selection and operation management for autonomous vehicles" refers to information including traffic data and analysis results that enable autonomous vehicles to operate efficiently and safely.

[0909] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[0910] composition

[0911] 1. Unmanned Aerial Vehicles (HAPS):

[0912] HAPS flies overhead and captures video data of a specific area in real time.

[0913] 2. Server:

[0914] The server has the function of receiving video data transmitted from HAPS.

[0915] The server sends the received video data to an analysis center.

[0916] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0917] The server generates and stores reports based on the classified data.

[0918] The server generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0919] 3. Analysis Center:

[0920] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0921] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0922] 4. Terminal:

[0923] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0924] The terminal displays operational management information for autonomous vehicles and is used for route selection.

[0925] operation

[0926] 1. Server receives video from HAPS:

[0927] The server receives real-time video data transmitted by the HAPS every hour. For example, it can acquire video data of a specific intersection or a wide area of ​​a city. The communication protocol is 5G or LoRaWAN.

[0928] 2. The server sends the data to the analysis center:

[0929] The server sends the received video data to an analysis center, where it uses generative artificial intelligence (e.g., OpenAI's GPT-4) to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0930] 3. The server receives, classifies, and saves the analysis results:

[0931] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) The classified data is stored in Amazon AWS S3.

[0932] 4. Server generates reports and traffic control information:

[0933] Using the classified data, the server automatically generates reports detailing the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[0934] In addition, it generates and provides information to optimize route selection and operation management for autonomous vehicles.

[0935] 5. Displaying reports and traffic control information on the device:

[0936] Users can access the interface using their devices to view reports and traffic management information generated from the server's database. Users can select reports for specific dates, times, and locations, and use them to check traffic information and plan routes for autonomous vehicles.

[0937] Examples of concrete examples and prompts

[0938] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users access the system from their devices and can view detailed traffic data for that intersection for a specific date and time in a report, while also receiving suggestions for optimal routes for autonomous vehicles.

[0939] An example of a new prompt is:

[0940] "Analyze real-time video data sent from an unmanned aerial vehicle (HAPS) and extract information on traffic volume, vehicle types, and pedestrian attributes at specific intersections. Based on the results, generate a Python program that suggests an optimal route with minimal risk of delay."

[0941] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[0942] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0943] Step 1:

[0944] The server receives video data from the HAPS. The input is real-time video data sent from the HAPS, and the output is video data stored in the server. The server receives this data using communication protocols such as 5G or LoRaWAN.

[0945] Step 2:

[0946] The server sends the received video data to the analysis center. The input is the video data stored on the server, and the output is the video data sent to the analysis center. The server transfers the data to the analysis center via the network.

[0947] Step 3:

[0948] The analysis center uses generative artificial intelligence (such as GPT-4) to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input is the video data sent to the analysis center, and the output is analyzed traffic information data. The generative artificial intelligence analyzes the video data and extracts specific patterns.

[0949] Step 4:

[0950] The server receives the analysis results from the analysis center. The input is the analysis result data sent from the analysis center, and the output is the analysis result data stored in the server. The server receives and stores the data from the analysis center.

[0951] Step 5:

[0952] The server categorizes the analyzed information and stores it in a database. The input is the analysis result data stored on the server, and the output is the categorized data stored in the database. The server organizes traffic volume, vehicle type, and pedestrian attribute information by category.

[0953] Step 6:

[0954] The server generates reports based on the classification data stored in the database. The input is the classification data in the database and the output is the generated report. The server automatically generates reports based on the classified data.

[0955] Step 7:

[0956] The server generates information to optimize route selection and operation management for autonomous vehicles. The input is the classification data in the database and the generated report, and the output is optimized operation information. The server proposes the optimal route taking into account real-time traffic information.

[0957] Step 8:

[0958] The terminal displays the reports and traffic control information sent from the server. The input is the reports and traffic control information sent from the server, and the output is the information displayed on the terminal screen. The user can check detailed traffic conditions and traffic information through the terminal interface.

[0959] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0960] This invention combines a system that analyzes traffic information based on video data from unmanned aerial vehicles (HAPS) and automatically generates user-friendly reports with an emotion engine that recognizes user emotions. This system receives video data from unmanned aerial vehicles, analyzes it at an analysis center, generates reports, and reflects user feedback through emotion recognition when providing the reports in a user-friendly format.

[0961] composition

[0962] 1. Unmanned aerial vehicle (HAPS)

[0963] HAPS flies overhead and captures video data of a specific area in real time.

[0964] 2. Server

[0965] The server has the function of receiving video data transmitted from HAPS.

[0966] The server sends the received video data to an analysis center.

[0967] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[0968] The server generates and stores reports based on the classified data.

[0969] 3. Analysis Center

[0970] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[0971] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[0972] 4. Terminal

[0973] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[0974] The device has a built-in emotion engine that recognizes the user's emotions and collects the user's emotion data.

[0975] operation

[0976] 1. The server receives the video from HAPS

[0977] The server receives and temporarily stores real-time video data transmitted by HAPS, which can capture images of the intersection or a wide area of ​​the city.

[0978] 2. The server sends the data to the analysis center

[0979] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[0980] 3. The server receives, classifies, and saves the analysis results

[0981] After receiving the analysis result data from the analysis center, the server classifies the results into each data category and stores them in a database.

[0982] 4. The server generates the report

[0983] Using the classified data, the server automatically generates reports detailing traffic volume fluctuations during specific times of the day, vehicle distribution, pedestrian attributes, and more.

[0984] 5. Viewing reports on your device

[0985] By accessing the interface using a terminal, a user can view reports generated from the server's database. The terminal uses an emotion engine to collect emotional data from the user's facial expressions, voice, movements, etc., and adjusts the display of the report based on that data.

[0986] 6. User Emotion Recognition and Feedback

[0987] The emotion engine analyzes the user's emotional data, and if it determines that the content of a report is difficult to understand, for example, it automatically reflects the feedback, such as revising the content or providing additional explanations.

[0988] Specific examples

[0989] For example, to analyze traffic data at a major intersection in a city, HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server then receives the analysis results, classifies and saves them into categories, and automatically generates reports.

[0990] Users access the system from their devices and view detailed traffic data for a specific intersection for a specific date and time as a report. The device's emotion engine recognizes the user's emotions. For example, if the user is viewing the report with interest, the system will suggest related information or data for other time periods. Conversely, if the user looks confused, the system will respond by providing more detailed information in the report.

[0991] In this way, the present invention can provide highly accurate and wide-ranging traffic information in real time, which can significantly improve the efficiency of urban planning and traffic management. Furthermore, by recognizing users' emotions and reflecting their feedback, a more user-friendly system can be realized.

[0992] The processing flow will be explained below.

[0993] Step 1:

[0994] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[0995] Step 2:

[0996] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[0997] Step 3:

[0998] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[0999] Step 4:

[1000] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[1001] Step 5:

[1002] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[1003] Step 6:

[1004] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[1005] Step 7:

[1006] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[1007] Step 8:

[1008] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[1009] Step 9:

[1010] The device's built-in emotion engine recognizes the user's emotions. When the user views a report, the emotion engine analyzes the user's facial expressions, voice, and movements to recognize the user's emotions in real time.

[1011] Step 10:

[1012] The device uses an emotion engine to analyze the user's emotional data. For example, if the user shows a confused expression, the device will display additional explanations. Conversely, if the user shows interest, the device will suggest related information or other reports.

[1013] Step 11:

[1014] The user reviews the provided reports and suggested information and obtains the information they need. They can then request further details or provide feedback.

[1015] In this way, the present invention realizes a system that recognizes the user's emotions and appropriately reflects feedback to provide reports and information that are easier to use and understand.

[1016] Example 2

[1017] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] Current traffic information analysis systems have limitations in efficiently analyzing wide-area video data acquired from unmanned aerial vehicles and providing it in a user-friendly format. Furthermore, they are unable to recognize user emotions in real time and incorporate feedback, limiting the extent to which they can improve usability. This creates problems that limit the efficiency of urban planning and traffic management.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1020] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and storing it in a database, means for generating and storing reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for the terminal to recognize user emotions and collect emotion data, and means for adjusting the display of the report based on the emotion data. This makes it possible to efficiently and comprehensively analyze traffic information and provide it in a user-friendly format. Furthermore, by recognizing user emotions in real time and reflecting feedback based on that, an even easier-to-use system can be realized.

[1021] An "unmanned aerial vehicle" is an aircraft that flies at high altitudes for long periods of time and captures video data of a specific area.

[1022] "Video data" refers to data containing a wide range of visual information captured by an unmanned aerial vehicle.

[1023] "Server" refers to a device that receives, stores, transmits, and classifies, stores, and analyzes video data.

[1024] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[1025] "Generative artificial intelligence" is a technology that uses machine learning models to analyze data and generate new information.

[1026] "Traffic volume" is data that indicates the frequency and density of traffic observed at a specific time or location.

[1027] "Vehicle type" is information indicating the type of vehicle contained in the analyzed video data.

[1028] "Pedestrian attribute information" is data that indicates specific characteristics of a pedestrian, such as gender and age.

[1029] "Categorizing" is the process of organizing and storing analyzed information according to specific categories.

[1030] A "database" is a system for efficiently storing and managing large amounts of data.

[1031] A "report" is a document that summarizes specific information in an easy-to-understand manner, generated based on classified data.

[1032] A "user-friendly format" is a presentation format that provides information in a way that is easy for users to understand and use.

[1033] A "terminal" is a device that has an interface that allows a user to access the system and perform operations and checks.

[1034] "Emotion recognition" is the process of analyzing a user's emotions from their facial expressions and voice.

[1035] "Emotional data" is specific information about the analyzed user's emotions.

[1036] "Feedback" is the process by which a system adjusts the information and functionality it displays based on the user's emotional data.

[1037] This invention combines a system that analyzes traffic information based on video data from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports, with an emotion engine that recognizes the user's emotions. This system is implemented in the following form.

[1038] First, unmanned aerial vehicles (HAPS) fly over a specific area at high altitude for a long period of time, capturing wide-area video data in real time. This video data includes high-resolution video of 1080p. The unmanned aerial vehicles transmit this video data to a server in real time using RTSP (Real-Time Streaming Protocol). A specific example would be monitoring a major intersection or a city square.

[1039] The server receives this video data and temporarily stores it. The server is equipped with a high-performance network interface card (NIC) and large-capacity storage. The received video data is sent to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer, enabling secure and high-speed data transfer. An example of a specific prompt sent by the server is as follows:

[1040] Destination URL: https: / / analysis-center.example.com / analyze

[1041] Sending data: Video data batch

[1042] Sending method: POST

[1043] The analysis center uses computers equipped with high-performance GPUs to analyze video data using generative artificial intelligence (AI). Specifically, TensorFlow's object detection API and PyTorch are used to extract attribute information on traffic volume, vehicle types, and pedestrians in the video. This allows for detailed analysis of hourly traffic volume, the proportion of vehicle types, and the gender and age of pedestrians.

[1044] The server receives the analysis results sent from the analysis center and classifies and stores them in a database. MySQL or PostgreSQL are commonly used relational databases for servers. By classifying and storing the analysis results by specific categories, the server can easily search and reference them later. Specific examples include "traffic volume data at intersection A" and "vehicle type data at intersection A."

[1045] Based on the classified data, the server automatically generates reports using Python data analysis and visualization libraries such as Pandas and Matplotlib, including line graphs showing the increase or decrease in traffic volume during specific times of day and pie charts showing the breakdown of vehicle types.

[1046] Users access reports using a terminal. Front-end frameworks such as React and Angular are used on the terminal to provide a user-friendly interface. When displaying reports, the terminal uses a built-in emotion engine to capture and analyze the user's facial expressions and voice in real time. This analysis is facilitated by the OpenCV library and IBM Watson's emotion analysis API. By analyzing the user's emotions, the system reflects the feedback in the report and adjusts the display content. For example, if the user shows a confused expression, the system automatically adds a supplementary explanation.

[1047] As described above, this invention can efficiently analyze a wide range of traffic information and provide user-friendly reports. Furthermore, by recognizing the user's emotions in real time and incorporating their feedback, it is possible to realize an even easier-to-use system.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1:

[1050] The server receives video from the unmanned aerial vehicle (HAPS). The server uses the RTSP protocol to receive 1080p high-resolution video data transmitted from the unmanned aerial vehicle in real time and temporarily stores it. The input is the video data from the unmanned aerial vehicle, and the output is the temporarily stored video data. Specifically, the server makes full use of a network interface card (NIC) to receive data at high speed and with stability.

[1051] Step 2:

[1052] The server sends the data to the analysis center. The server then sends the temporarily stored video data to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer. The input is the temporarily stored video data, and the output is the video data sent to the analysis center. Specifically, the server batches the data and generates a prompt to send to the analysis center.

[1053] Step 3:

[1054] The analysis center analyzes the video data. The analysis center uses a computer equipped with a high-performance GPU to analyze the video data using a generative AI model. The input is the video data sent to the analysis center, and the output is analyzed traffic information, vehicle information, and pedestrian attribute information. Specifically, a machine learning model using TensorFlow and PyTorch detects vehicles and pedestrians in the video and analyzes their characteristics.

[1055] Step 4:

[1056] The server receives, classifies, and stores the analysis results. The server receives the analysis results sent from the analysis center and classifies and stores them in a relational database such as MySQL or PostgreSQL. The input is the analysis results from the analysis center, and the output is the analysis results that have been classified and stored in the database. Specifically, the server classifies the results into categories such as "traffic volume data," "vehicle type data," and "pedestrian attribute data."

[1057] Step 5:

[1058] The server generates the report. Based on the classified data, the server automatically generates the report using Python's Pandas and Matplotlib. The input is the classified data stored in the database, and the output is the generated report. Specifically, the server creates a line graph showing the increase or decrease in traffic volume during a specific time period, and a pie chart showing the proportion of vehicle types, and compiles these into a report.

[1059] Step 6:

[1060] The terminal displays the report. The terminal uses a front-end framework such as React or Angular to provide a user-friendly interface. The input is the generated report, and the output is the report displayed to the user. Specifically, the terminal updates the report display according to user operations.

[1061] Step 7:

[1062] The device recognizes the user's emotions and provides feedback. The device uses a built-in emotion engine to analyze the user's facial expressions and voice in real time. The input is the user's facial and voice data, and the output is analyzed emotion data. Specifically, the OpenCV library and IBM Watson emotion analysis API analyze the user's emotions and reflects the feedback in the report based on that. For example, if the user is confused, additional explanations will be displayed.

[1063] (Application example 2)

[1064] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] Conventional traffic information analysis systems can provide highly accurate traffic information in real time, but they lack feedback and display adjustments based on the user's emotional state. Furthermore, they lack flexible adjustments to avoid user stress and confusion, making it difficult to provide a user-friendly interface. Furthermore, it is necessary to recognize the emotions of drivers and passengers and provide optimal navigation and information display based on those emotions.

[1066] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving a report based on the categorized data, means for providing the generated report in a user-friendly format, means for recognizing the user's emotions using sensors in the vehicle, and means for adjusting the display or guidance based on the emotion recognition data. This enables the provision of real-time traffic information according to the user's emotions and safe and comfortable navigation.

[1067] An "unmanned aerial vehicle" is an aircraft that does not have a human pilot and that flies autonomously or by remote control.

[1068] "Video data" refers to data that includes visual information acquired using a camera, sensor, etc.

[1069] "Analysis Center" means a facility or system that processes and analyzes received data.

[1070] "Generative AI" is an AI that has the ability to learn large amounts of data and analyze and generate data that is given to it.

[1071] "Traffic volume" is an indicator that indicates the number of vehicles or people passing through a specific location within a certain period of time.

[1072] "Vehicle type" is a classification used to distinguish between different types of automobiles.

[1073] "Attribute information of pedestrians" is information including characteristics of pedestrians such as gender, age, and clothing.

[1074] A "database" is a system for efficiently storing, searching, and managing data.

[1075] "Report" means a report or summary of information generated based on analyzed data.

[1076] "User-friendly" is a term that describes something that is easy for users to use and understand.

[1077] A "sensor" is a device that detects a physical quantity (e.g., temperature, humidity, movement, etc.) and converts it into data.

[1078] "Emotion recognition data" is data used to identify emotions from human facial expressions, voice, movements, etc.

[1079] "Display adaptation" refers to changing how information is displayed in response to user factors.

[1080] "Guidance" is a term that means providing guidance to a destination or information.

[1081] The system that specifically embodies this invention is composed of an unmanned aerial vehicle (HAPS), a server, an analysis center, a database, and terminals and sensors within the autonomous vehicle.

[1082] The server first receives video data acquired from an unmanned aerial vehicle (HAPS). The unmanned aerial vehicle is capable of capturing images of a wide area and transmitting real-time video data. The received video data is temporarily stored by the server and then sent to the analysis center.

[1083] The analysis center uses generative artificial intelligence to analyze the video data. Specifically, information on traffic volume, vehicle types, and pedestrian attributes is obtained. Examples of generative artificial intelligence used include OpenAI's GPT and Google's DeepMind.

[1084] The analyzed data is sent to a server, which categorizes it and stores it in a database. The database can be MySQL or PostgreSQL. The server then automatically generates and stores reports based on the categorized data.

[1085] The devices inside the autonomous vehicles are equipped with cameras and microphones that detect the emotions of the driver and passengers. Emotion recognition engines such as Microsoft Azure Cognitive Services and Amazon Rekognition are used for emotion recognition. Emotion data is sent to a server, which then adjusts the display and guidance based on this data.

[1086] As a concrete example, while an autonomous vehicle is driving in an urban area, HAPS captures real-time images of the surrounding traffic conditions. This image data is sent to a server and then analyzed as traffic information at an analysis center. The captured traffic information is then sent to the server and stored in a database. If the driver is feeling stressed, the sensor sends this emotional data to the server, which then generates a report suggesting the optimal detour route. The report is then displayed on the device, providing the driver with comfortable navigation.

[1087] An example prompt is:

[1088] "Explain the system that analyzes video data from HAPS, obtains traffic information, and provides the optimal navigation route based on the driver's emotional data."

[1089] This will enable the provision of real-time traffic information that responds to the user's emotions and safe and comfortable navigation.

[1090] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1091] Step 1:

[1092] The server receives video data from the unmanned aerial vehicle. The HAPS captures a wide area and transmits real-time video data. This video data is input to the server and temporarily stored. The input data is stored in temporary storage and then goes to the next analysis step.

[1093] Step 2:

[1094] The server sends the received video data to the analysis center. Using a pre-configured communication protocol, the server transfers the stored video data to the analysis center. The input data is the video data from HAPS, and the output data is the video data transferred to the analysis center.

[1095] Step 3:

[1096] The analysis center uses generative artificial intelligence to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input data is the video data sent from HAPS, and a generative AI model (e.g., OpenAI's GPT or Google's DeepMind) is used. The analyzed data is extracted, and traffic information is obtained as output data. The obtained information includes the number of vehicles, vehicle type identification, and pedestrian attributes (gender, age, etc.).

[1097] Step 4:

[1098] The server receives traffic information sent from the analysis center, categorizes it into categories, and stores it in a database. The input data is the traffic information sent from the analysis center, and the output data is database entries organized by category. The server does this using a database management system (e.g., MySQL, PostgreSQL).

[1099] Step 5:

[1100] The server automatically generates and saves reports based on the classified data. The generated reports include information such as increases or decreases in traffic volume during specific time periods, vehicle distribution, and pedestrian attributes. The input data is the categorized traffic information stored in the database, and the output data is the generated report.

[1101] Step 6:

[1102] The terminal inside the autonomous vehicle displays the report and recognizes the user's emotions using sensors inside the vehicle. An emotion recognition engine (e.g., Microsoft Azure Cognitive Services, Amazon Rekognition) is used, and the input data is emotion information obtained from the camera and microphone. The output data is the analyzed emotion data.

[1103] Step 7:

[1104] The server adjusts the display and guidance based on emotion recognition data. The input data is emotion data sent from the emotion recognition engine, and the output data is adjusted navigation and information display. For example, if the driver is feeling stressed, the server will optimize the route and suggest a more comfortable route.

[1105] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1107] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1108] [Fourth embodiment]

[1109] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1110] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1112] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1113] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1116] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1117] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1118] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1119] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1120] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1121] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1122] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[1123] composition

[1124] 1. Unmanned aerial vehicle (HAPS)

[1125] HAPS flies overhead and captures video data of a specific area in real time.

[1126] 2. Server

[1127] The server has the function of receiving video data transmitted from HAPS.

[1128] The server sends the received video data to an analysis center.

[1129] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[1130] The server generates and stores reports based on the classified data.

[1131] 3. Analysis Center

[1132] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[1133] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[1134] 4. Terminal

[1135] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[1136] operation

[1137] 1. The server receives the video from HAPS

[1138] Every hour, the server receives real-time video data transmitted by HAPS, allowing it to obtain video data for a specific intersection or a wide area of ​​a city.

[1139] 2. The server sends the data to the analysis center

[1140] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[1141] 3. The server receives, classifies, and saves the analysis results

[1142] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) and stores the classified data in a database.

[1143] 4. The server generates the report

[1144] Using the classified data, the server automatically generates reports detailing, for example, the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[1145] 5. Viewing reports on your device

[1146] Users can access the interface using their devices to view reports generated from the server's database. Users can select reports for specific dates, times, and locations, and check and analyze traffic information.

[1147] Specific examples

[1148] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection over several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time in the form of a report.

[1149] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[1150] The processing flow will be explained below.

[1151] Step 1:

[1152] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[1153] Step 2:

[1154] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[1155] Step 3:

[1156] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[1157] Step 4:

[1158] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[1159] Step 5:

[1160] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[1161] Step 6:

[1162] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[1163] Step 7:

[1164] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[1165] Step 8:

[1166] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[1167] Step 9:

[1168] The user can view the report they have selected. The user can check the details of the selected report and obtain the information they need. This allows the user to easily use traffic analysis data for a specific time period or area.

[1169] Example 1

[1170] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1171] Conventional traffic information collection systems have struggled to collect and analyze a wide range of data in real time and provide it in a user-friendly report format. Furthermore, classifying various data and generating reports requires a huge amount of time and effort, making them inadequate for urban planning and traffic management, which require rapid response. Furthermore, displaying analysis results makes it difficult to provide detailed visual data tailored to specific points in time and locations.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1173] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, means for converting the format of the received video data, means for receiving analysis results from the analysis center and categorizing them into data categories, means for indexing the data stored in the database, means for generating reports based on templates, and means for accessing the reports from a terminal and displaying them at a specific date, time, or location. This enables the collection and analysis of wide-ranging and detailed traffic information in real time and the rapid provision of such information in a user-friendly format.

[1174] An "unmanned aerial vehicle" is an aircraft that flies under remote or automatic control and photographs and collects data over a specific area.

[1175] "Video data" refers to data containing visual information captured by unmanned aerial vehicles, capturing traffic conditions and people's movements.

[1176] An "analysis center" is a facility or system that analyzes received video data and extracts information such as traffic volume, vehicle types, and pedestrian attributes.

[1177] "Generative artificial intelligence" is an artificial intelligence model that has the ability to perform analysis of data and generate specific results.

[1178] "Classification by category" means dividing the acquired information based on specific attributes, such as traffic volume, vehicle type, and pedestrian attribute information.

[1179] A "database" is an information system for storing and managing classified data, enabling efficient searching and use of the data.

[1180] A "report" is a document or file that organizes analyzed data and presents it in a visually easy-to-understand format.

[1181] "User-friendly" means that the system or interface is designed and formatted to be easy for users to use.

[1182] "Format conversion" refers to the process of changing data into a format that is easy for the analysis center to process.

[1183] "Indexing" means adding identifiers or keys to data in a database to improve search efficiency.

[1184] A "template" is a predefined format or structure used to generate a report.

[1185] A "terminal" is a device used by a user to access the system and display data, examples of which include a PC or smartphone.

[1186] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle and automatically generates user-friendly reports. This system is configured as follows.

[1187] unmanned aerial vehicle

[1188] An unmanned aerial vehicle (HAPS) flies overhead and captures specific areas in real time. For example, a HAPS can capture images of major intersections in a city and send the video data to a server on the ground. The HAPS's flight altitude and camera resolution are set according to the situation, allowing it to capture detailed traffic conditions.

[1189] server

[1190] The server has the following functions:

[1191] 1. Video reception:

[1192] The server receives the video data sent from the HAPS via a monitoring port and stores it in temporary storage in real time. HTTP or FTP is used as the communication protocol.

[1193] 2. Data transmission:

[1194] The received video data is first converted into a format before being sent to the analysis center. For example, JPEG format images are converted into MP4 format videos and then sent to the analysis center.

[1195] 3. Receiving and classifying analysis results:

[1196] The analysis center returns the analysis results data, which includes information on traffic volume, vehicle types, and pedestrian attributes. The server categorizes this data and stores it in a database.

[1197] 4. Report Generation:

[1198] The server uses the classified data to automatically generate reports based on the specified template, which are saved in PDF or HTML format.

[1199] Analysis Center

[1200] The analysis center has the following functions:

[1201] 1. Data Analysis:

[1202] The received video data is analyzed using generative artificial intelligence (AI model), which calculates traffic volume, classifies vehicle types, and estimates the age and gender of pedestrians.

[1203] Terminal

[1204] Users access reports using a device (PC or smartphone).

[1205] 1. Access the interface:

[1206] Users access the server interface through a web browser or a dedicated app.

[1207] 2. Report Viewing:

[1208] Users can select reports for specific dates, times, and locations to view detailed traffic information, including, for example, traffic volume fluctuations during specific times, vehicle type distribution, and the gender and age distribution of pedestrians.

[1209] Specific examples

[1210] For example, consider analyzing traffic data at a major intersection in a city.

[1211] HAPS captures the intersection for several hours and sends the video data to a server. The server then sends the video to an analysis center, where it is analyzed using generative artificial intelligence. The analysis center analyzes traffic volume, vehicle type, pedestrian gender and age, and sends the results back to the server. The server categorizes and saves the data and automatically generates a report. Users can access the system from their devices and view detailed traffic data for that intersection for a specific date and time.

[1212] Prompt Sentence Examples

[1213] "Analyze traffic data for a specific area of ​​a specific city and generate a report with information on traffic volume, vehicle types, and pedestrian attributes. For example, I want to know the traffic conditions at major intersections from 8:00 to 9:00 AM."

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] Step 1:

[1216] The server receives the video data from HAPS

[1217] Input: Real-time video data transmitted from an unmanned aerial vehicle (HAPS).

[1218] Specific operation: The server listens on a specified communication port (e.g., HTTP or FTP) and receives real-time video data from HAPS. The received data is stored in temporary storage.

[1219] Output: Video data saved in temporary storage.

[1220] Step 2:

[1221] The server converts the format of the video data

[1222] Input: Video data stored in temporary storage.

[1223] Specific operation: The server converts the format of the received video data into a format acceptable to the analysis center (e.g., converting from JPEG to MP4). This process uses an image conversion library (e.g., FFmpeg).

[1224] Output: Format converted video data.

[1225] Step 3:

[1226] The server sends the converted data to the analysis center.

[1227] Input: Format-converted video data.

[1228] How it works: The server uses an API call, such as an HTTP POST request, to send the converted video data to the analysis center.

[1229] Output: Video data sent to analysis center.

[1230] Step 4:

[1231] The analysis center analyzes the video data

[1232] Input: Video data sent to analysis center.

[1233] How it works: The analysis center uses generative artificial intelligence to analyze the received video data, including calculating traffic volume, classifying vehicle types, and estimating pedestrian attributes (age and gender).

[1234] Output: Analyzed traffic volume data, vehicle type data, and pedestrian attribute information data.

[1235] Step 5:

[1236] The server receives, classifies, and stores the analysis results

[1237] Input: Analysis result data sent from the analysis center.

[1238] How it works: The server receives data sent from the analysis center and classifies it into categories such as traffic volume data, vehicle data, pedestrian data, etc. The classified data is indexed for efficient search and stored in a database.

[1239] Output: Data categorized and stored in a database.

[1240] Step 6:

[1241] The server generates the report

[1242] Input: Classified data stored in a database.

[1243] Specific operation: The server reads a prepared report template based on the classified data, embeds the data in the template, and generates a report. The report is generated in PDF or HTML format, and graphs and tables are added to fit each format.

[1244] Output: Generated report in PDF and HTML format.

[1245] Step 7:

[1246] Accessing reports from your device

[1247] Input: The query information for the user to access (date, time, location, etc.).

[1248] How it works: A user accesses the web interface provided by the server using a device such as a PC or smartphone, selects a report for a specific date, time, or location, and the server sends a query to the database to retrieve the corresponding report.

[1249] Output: The report displayed on the terminal.

[1250] Step 8:

[1251] Users view and analyze reports

[1252] Input: Report data retrieved from the server.

[1253] What it does: The user views a report on their device, which includes graphs of traffic volume fluctuations, vehicle type distribution, and the gender and age distribution of pedestrians.

[1254] Output: The results and insights of the user's analysis.

[1255] (Application example 1)

[1256] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1257] In modern cities, obtaining traffic information quickly and accurately and using that information to optimize route selection and operation management for autonomous vehicles is a critical issue. Conventional traffic information systems can only provide a limited range of information and lack real-time performance. Furthermore, manually analyzing and updating detailed traffic data requires a great deal of effort and time. There is a need to resolve these issues and build a system that provides highly accurate and rapid traffic information.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1259] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for generating and providing information for optimizing route selection and operation management of autonomous vehicles. This enables the efficient operation of autonomous vehicles by analyzing a wide range of traffic information obtained from the unmanned aerial vehicle in real time.

[1260] An "unmanned aerial vehicle" is an aircraft that is remotely controlled and operated using advanced technology, and is a device used to collect video data of a specific area.

[1261] "Video data" refers to digital data of real-time video of a specific area captured from the sky by an unmanned aerial vehicle.

[1262] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[1263] "Generative artificial intelligence" is an AI technology that uses machine learning techniques to find patterns in huge amounts of data and generate information that meets specific purposes.

[1264] "Traffic volume" is information indicating the number and density of vehicles traveling on a road during a specific time period.

[1265] "Vehicle type" is information indicating the type or classification of vehicles included in the traffic data.

[1266] "Pedestrian attribute information" is information that indicates characteristics of pedestrians, such as gender and age group, contained in the traffic data.

[1267] A "database" is an information management system that organizes and stores acquired information so that it can be retrieved as needed.

[1268] "Report" means an automatically generated summary of information based on analyzed and classified traffic data, in the form provided to the User.

[1269] A "user-friendly format" is a presentation of information that is designed to be easy for users to understand and operate.

[1270] "Information for optimizing route selection and operation management for autonomous vehicles" refers to information including traffic data and analysis results that enable autonomous vehicles to operate efficiently and safely.

[1271] This invention relates to a system that analyzes traffic information based on video data obtained from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports. This system is configured and operates as follows.

[1272] composition

[1273] 1. Unmanned Aerial Vehicles (HAPS):

[1274] HAPS flies overhead and captures video data of a specific area in real time.

[1275] 2. Server:

[1276] The server has the function of receiving video data transmitted from HAPS.

[1277] The server sends the received video data to an analysis center.

[1278] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[1279] The server generates and stores reports based on the classified data.

[1280] The server generates and provides information to optimize route selection and operation management for autonomous vehicles.

[1281] 3. Analysis Center:

[1282] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[1283] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[1284] 4. Terminal:

[1285] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[1286] The terminal displays operational management information for autonomous vehicles and is used for route selection.

[1287] operation

[1288] 1. Server receives video from HAPS:

[1289] The server receives real-time video data transmitted by the HAPS every hour. For example, it can acquire video data of a specific intersection or a wide area of ​​a city. The communication protocol is 5G or LoRaWAN.

[1290] 2. The server sends the data to the analysis center:

[1291] The server sends the received video data to an analysis center, where it uses generative artificial intelligence (e.g., OpenAI's GPT-4) to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[1292] 3. The server receives, classifies, and saves the analysis results:

[1293] After receiving the analysis result data from the analysis center, the server classifies it into each data category (traffic volume, vehicle type, pedestrian attribute information, etc.) The classified data is stored in Amazon AWS S3.

[1294] 4. Server generates reports and traffic control information:

[1295] Using the classified data, the server automatically generates reports detailing the increase or decrease in traffic volume at specific times of the day, the distribution of vehicle types, and the gender and age distribution of pedestrians.

[1296] In addition, it generates and provides information to optimize route selection and operation management for autonomous vehicles.

[1297] 5. Displaying reports and traffic control information on the device:

[1298] Users can access the interface using their devices to view reports and traffic management information generated from the server's database. Users can select reports for specific dates, times, and locations, and use them to check traffic information and plan routes for autonomous vehicles.

[1299] Examples of concrete examples and prompts

[1300] For example, consider the case of analyzing traffic data at a major intersection in a city. HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server receives the analysis results, classifies and saves them into categories, and automatically generates a report. Users access the system from their devices and can view detailed traffic data for that intersection for a specific date and time in a report, while also receiving suggestions for optimal routes for autonomous vehicles.

[1301] An example of a new prompt is:

[1302] "Analyze real-time video data sent from an unmanned aerial vehicle (HAPS) and extract information on traffic volume, vehicle types, and pedestrian attributes at specific intersections. Based on the results, generate a Python program that suggests an optimal route with minimal risk of delay."

[1303] In this way, the present invention makes it possible to provide highly accurate and wide-ranging traffic information in real time without requiring a huge amount of manpower and time, thereby significantly improving the efficiency of urban planning and traffic management.

[1304] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1305] Step 1:

[1306] The server receives video data from the HAPS. The input is real-time video data sent from the HAPS, and the output is video data stored in the server. The server receives this data using communication protocols such as 5G or LoRaWAN.

[1307] Step 2:

[1308] The server sends the received video data to the analysis center. The input is the video data stored on the server, and the output is the video data sent to the analysis center. The server transfers the data to the analysis center via the network.

[1309] Step 3:

[1310] The analysis center uses generative artificial intelligence (such as GPT-4) to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input is the video data sent to the analysis center, and the output is analyzed traffic information data. The generative artificial intelligence analyzes the video data and extracts specific patterns.

[1311] Step 4:

[1312] The server receives the analysis results from the analysis center. The input is the analysis result data sent from the analysis center, and the output is the analysis result data stored in the server. The server receives and stores the data from the analysis center.

[1313] Step 5:

[1314] The server categorizes the analyzed information and stores it in a database. The input is the analysis result data stored on the server, and the output is the categorized data stored in the database. The server organizes traffic volume, vehicle type, and pedestrian attribute information by category.

[1315] Step 6:

[1316] The server generates reports based on the classification data stored in the database. The input is the classification data in the database and the output is the generated report. The server automatically generates reports based on the classified data.

[1317] Step 7:

[1318] The server generates information to optimize route selection and operation management for autonomous vehicles. The input is the classification data in the database and the generated report, and the output is optimized operation information. The server proposes the optimal route taking into account real-time traffic information.

[1319] Step 8:

[1320] The terminal displays the reports and traffic control information sent from the server. The input is the reports and traffic control information sent from the server, and the output is the information displayed on the terminal screen. The user can check detailed traffic conditions and traffic information through the terminal interface.

[1321] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1322] This invention combines a system that analyzes traffic information based on video data from unmanned aerial vehicles (HAPS) and automatically generates user-friendly reports with an emotion engine that recognizes user emotions. This system receives video data from unmanned aerial vehicles, analyzes it at an analysis center, generates reports, and reflects user feedback through emotion recognition when providing the reports in a user-friendly format.

[1323] composition

[1324] 1. Unmanned aerial vehicle (HAPS)

[1325] HAPS flies overhead and captures video data of a specific area in real time.

[1326] 2. Server

[1327] The server has the function of receiving video data transmitted from HAPS.

[1328] The server sends the received video data to an analysis center.

[1329] The server receives the analysis results from the analysis center and has the function of classifying and storing them.

[1330] The server generates and stores reports based on the classified data.

[1331] 3. Analysis Center

[1332] The analysis center is a facility or system that analyzes received video data using generative artificial intelligence.

[1333] The analysis center analyzes traffic volume, vehicle types, pedestrian attribute information, etc. and sends the results back to the server.

[1334] 4. Terminal

[1335] The terminal has a user-accessible interface and is capable of displaying reports sent from the server.

[1336] The device has a built-in emotion engine that recognizes the user's emotions and collects the user's emotion data.

[1337] operation

[1338] 1. The server receives the video from HAPS

[1339] The server receives and temporarily stores real-time video data transmitted by HAPS, which can capture images of the intersection or a wide area of ​​the city.

[1340] 2. The server sends the data to the analysis center

[1341] The received video data is sent to an analysis center, which uses generative artificial intelligence to analyze the video data and extract information on traffic volume, vehicle types, and pedestrian attributes.

[1342] 3. The server receives, classifies, and saves the analysis results

[1343] After receiving the analysis result data from the analysis center, the server classifies the results into each data category and stores them in a database.

[1344] 4. The server generates the report

[1345] Using the classified data, the server automatically generates reports detailing traffic volume fluctuations during specific times of the day, vehicle distribution, pedestrian attributes, and more.

[1346] 5. Viewing reports on your device

[1347] By accessing the interface using a terminal, a user can view reports generated from the server's database. The terminal uses an emotion engine to collect emotional data from the user's facial expressions, voice, movements, etc., and adjusts the display of the report based on that data.

[1348] 6. User Emotion Recognition and Feedback

[1349] The emotion engine analyzes the user's emotional data, and if it determines that the content of a report is difficult to understand, for example, it automatically reflects the feedback, such as revising the content or providing additional explanations.

[1350] Specific examples

[1351] For example, to analyze traffic data at a major intersection in a city, HAPS captures footage of the intersection for several hours and sends the video data to a server. The server then sends the footage to an analysis center, which analyzes traffic volume, vehicle types, and the gender and age of pedestrians. The server then receives the analysis results, classifies and saves them into categories, and automatically generates reports.

[1352] Users access the system from their devices and view detailed traffic data for a specific intersection for a specific date and time as a report. The device's emotion engine recognizes the user's emotions. For example, if the user is viewing the report with interest, the system will suggest related information or data for other time periods. Conversely, if the user looks confused, the system will respond by providing more detailed information in the report.

[1353] In this way, the present invention can provide highly accurate and wide-ranging traffic information in real time, which can significantly improve the efficiency of urban planning and traffic management. Furthermore, by recognizing users' emotions and reflecting their feedback, a more user-friendly system can be realized.

[1354] The processing flow will be explained below.

[1355] Step 1:

[1356] The server receives the video data from HAPS. Specifically, the server specifies the real-time video stream URL sent by HAPS and receives the video data from that URL. The received video data is temporarily stored on the server.

[1357] Step 2:

[1358] The server sends the received video data to the analysis center. The server then sends the temporarily stored video data to the analysis center's API endpoint and issues an analysis request. At this stage, the server waits for a response confirming that the data was sent correctly.

[1359] Step 3:

[1360] The analysis center analyzes the video data using generative artificial intelligence. The analysis center analyzes the received video data and extracts information on traffic volume, vehicle types, and pedestrian attributes. Once the analysis results are ready, they are sent back to the server.

[1361] Step 4:

[1362] The server receives the analysis result data from the analysis center. The server checks the received analysis result data to verify that there are no omissions or errors. After the verification is complete, the server proceeds to the next step.

[1363] Step 5:

[1364] The server classifies the analysis results data and saves it in a database. Specifically, the server classifies the analysis results into categories such as traffic volume, vehicle type, and pedestrian attribute information. The classified data is stored in a database.

[1365] Step 6:

[1366] The server generates reports based on the classified data. The server retrieves the necessary data from the database and automatically generates easy-to-read reports in a user-friendly format. The reports are saved as files.

[1367] Step 7:

[1368] The server provides the generated reports to the user interface, and the server sends the report data to the front-end interface for user access.

[1369] Step 8:

[1370] The user accesses a dedicated interface from a terminal, through which the user can check a list of reports provided by the server and select a specific report.

[1371] Step 9:

[1372] The device's built-in emotion engine recognizes the user's emotions. When the user views a report, the emotion engine analyzes the user's facial expressions, voice, and movements to recognize the user's emotions in real time.

[1373] Step 10:

[1374] The device uses an emotion engine to analyze the user's emotional data. For example, if the user shows a confused expression, the device will display additional explanations. Conversely, if the user shows interest, the device will suggest related information or other reports.

[1375] Step 11:

[1376] The user reviews the provided reports and suggested information and obtains the information they need. They can then request further details or provide feedback.

[1377] In this way, the present invention realizes a system that recognizes the user's emotions and appropriately reflects feedback to provide reports and information that are easier to use and understand.

[1378] Example 2

[1379] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1380] Current traffic information analysis systems have limitations in efficiently analyzing wide-area video data acquired from unmanned aerial vehicles and providing it in a user-friendly format. Furthermore, they are unable to recognize user emotions in real time and incorporate feedback, limiting the extent to which they can improve usability. This creates problems that limit the efficiency of urban planning and traffic management.

[1381] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1382] In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center to obtain traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the obtained information and storing it in a database, means for generating and storing reports based on the categorized data, means for providing the generated reports in a user-friendly format, and means for the terminal to recognize user emotions and collect emotion data, and means for adjusting the display of the report based on the emotion data. This makes it possible to efficiently and comprehensively analyze traffic information and provide it in a user-friendly format. Furthermore, by recognizing user emotions in real time and reflecting feedback based on that, an even easier-to-use system can be realized.

[1383] An "unmanned aerial vehicle" is an aircraft that flies at high altitudes for long periods of time and captures video data of a specific area.

[1384] "Video data" refers to data containing a wide range of visual information captured by an unmanned aerial vehicle.

[1385] "Server" refers to a device that receives, stores, transmits, and classifies, stores, and analyzes video data.

[1386] An "analysis center" is a facility or system that uses generative artificial intelligence to analyze received video data and extract traffic information.

[1387] "Generative artificial intelligence" is a technology that uses machine learning models to analyze data and generate new information.

[1388] "Traffic volume" is data that indicates the frequency and density of traffic observed at a specific time or location.

[1389] "Vehicle type" is information indicating the type of vehicle contained in the analyzed video data.

[1390] "Pedestrian attribute information" is data that indicates specific characteristics of a pedestrian, such as gender and age.

[1391] "Categorizing" is the process of organizing and storing analyzed information according to specific categories.

[1392] A "database" is a system for efficiently storing and managing large amounts of data.

[1393] A "report" is a document that summarizes specific information in an easy-to-understand manner, generated based on classified data.

[1394] A "user-friendly format" is a presentation format that provides information in a way that is easy for users to understand and use.

[1395] A "terminal" is a device that has an interface that allows a user to access the system and perform operations and checks.

[1396] "Emotion recognition" is the process of analyzing a user's emotions from their facial expressions and voice.

[1397] "Emotional data" is specific information about the analyzed user's emotions.

[1398] "Feedback" is the process by which a system adjusts the information and functionality it displays based on the user's emotional data.

[1399] This invention combines a system that analyzes traffic information based on video data from an unmanned aerial vehicle (HAPS) and automatically generates user-friendly reports, with an emotion engine that recognizes the user's emotions. This system is implemented in the following form.

[1400] First, unmanned aerial vehicles (HAPS) fly over a specific area at high altitude for a long period of time, capturing wide-area video data in real time. This video data includes high-resolution video of 1080p. The unmanned aerial vehicles transmit this video data to a server in real time using RTSP (Real-Time Streaming Protocol). A specific example would be monitoring a major intersection or a city square.

[1401] The server receives this video data and temporarily stores it. The server is equipped with a high-performance network interface card (NIC) and large-capacity storage. The received video data is sent to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer, enabling secure and high-speed data transfer. An example of a specific prompt sent by the server is as follows:

[1402] Destination URL: https: / / analysis-center.example.com / analyze

[1403] Sending data: Video data batch

[1404] Sending method: POST

[1405] The analysis center uses computers equipped with high-performance GPUs to analyze video data using generative artificial intelligence (AI). Specifically, TensorFlow's object detection API and PyTorch are used to extract attribute information on traffic volume, vehicle types, and pedestrians in the video. This allows for detailed analysis of hourly traffic volume, the proportion of vehicle types, and the gender and age of pedestrians.

[1406] The server receives the analysis results sent from the analysis center and classifies and stores them in a database. MySQL or PostgreSQL are commonly used relational databases for servers. By classifying and storing the analysis results by specific categories, the server can easily search and reference them later. Specific examples include "traffic volume data at intersection A" and "vehicle type data at intersection A."

[1407] Based on the classified data, the server automatically generates reports using Python data analysis and visualization libraries such as Pandas and Matplotlib, including line graphs showing the increase or decrease in traffic volume during specific times of day and pie charts showing the breakdown of vehicle types.

[1408] Users access reports using a terminal. Front-end frameworks such as React and Angular are used on the terminal to provide a user-friendly interface. When displaying reports, the terminal uses a built-in emotion engine to capture and analyze the user's facial expressions and voice in real time. This analysis is facilitated by the OpenCV library and IBM Watson's emotion analysis API. By analyzing the user's emotions, the system reflects the feedback in the report and adjusts the display content. For example, if the user shows a confused expression, the system automatically adds a supplementary explanation.

[1409] As described above, this invention can efficiently analyze a wide range of traffic information and provide user-friendly reports. Furthermore, by recognizing the user's emotions in real time and incorporating their feedback, it is possible to realize an even easier-to-use system.

[1410] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1411] Step 1:

[1412] The server receives video from the unmanned aerial vehicle (HAPS). The server uses the RTSP protocol to receive 1080p high-resolution video data transmitted from the unmanned aerial vehicle in real time and temporarily stores it. The input is the video data from the unmanned aerial vehicle, and the output is the temporarily stored video data. Specifically, the server makes full use of a network interface card (NIC) to receive data at high speed and with stability.

[1413] Step 2:

[1414] The server sends the data to the analysis center. The server then sends the temporarily stored video data to the analysis center using the HTTPS protocol. Cloud storage such as Amazon S3 is used for data transfer. The input is the temporarily stored video data, and the output is the video data sent to the analysis center. Specifically, the server batches the data and generates a prompt to send to the analysis center.

[1415] Step 3:

[1416] The analysis center analyzes the video data. The analysis center uses a computer equipped with a high-performance GPU to analyze the video data using a generative AI model. The input is the video data sent to the analysis center, and the output is analyzed traffic information, vehicle information, and pedestrian attribute information. Specifically, a machine learning model using TensorFlow and PyTorch detects vehicles and pedestrians in the video and analyzes their characteristics.

[1417] Step 4:

[1418] The server receives, classifies, and stores the analysis results. The server receives the analysis results sent from the analysis center and classifies and stores them in a relational database such as MySQL or PostgreSQL. The input is the analysis results from the analysis center, and the output is the analysis results that have been classified and stored in the database. Specifically, the server classifies the results into categories such as "traffic volume data," "vehicle type data," and "pedestrian attribute data."

[1419] Step 5:

[1420] The server generates the report. Based on the classified data, the server automatically generates the report using Python's Pandas and Matplotlib. The input is the classified data stored in the database, and the output is the generated report. Specifically, the server creates a line graph showing the increase or decrease in traffic volume during a specific time period, and a pie chart showing the proportion of vehicle types, and compiles these into a report.

[1421] Step 6:

[1422] The terminal displays the report. The terminal uses a front-end framework such as React or Angular to provide a user-friendly interface. The input is the generated report, and the output is the report displayed to the user. Specifically, the terminal updates the report display according to user operations.

[1423] Step 7:

[1424] The device recognizes the user's emotions and provides feedback. The device uses a built-in emotion engine to analyze the user's facial expressions and voice in real time. The input is the user's facial and voice data, and the output is analyzed emotion data. Specifically, the OpenCV library and IBM Watson emotion analysis API analyze the user's emotions and reflects the feedback in the report based on that. For example, if the user is confused, additional explanations will be displayed.

[1425] (Application example 2)

[1426] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] Conventional traffic information analysis systems can provide highly accurate traffic information in real time, but they lack feedback and display adjustments based on the user's emotional state. Furthermore, they lack flexible adjustments to avoid user stress and confusion, making it difficult to provide a user-friendly interface. Furthermore, it is necessary to recognize the emotions of drivers and passengers and provide optimal navigation and information display based on those emotions.

[1428] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data from the unmanned aerial vehicle, means for transmitting the received video data to an analysis center, means for analyzing the video data using generative artificial intelligence at the analysis center and acquiring traffic volume, vehicle type, and pedestrian attribute information, means for categorizing the acquired information and saving it in a database, means for generating and saving a report based on the categorized data, means for providing the generated report in a user-friendly format, means for recognizing the user's emotions using sensors in the vehicle, and means for adjusting the display or guidance based on the emotion recognition data. This enables the provision of real-time traffic information according to the user's emotions and safe and comfortable navigation.

[1429] An "unmanned aerial vehicle" is an aircraft that does not have a human pilot and that flies autonomously or by remote control.

[1430] "Video data" refers to data that includes visual information acquired using a camera, sensor, etc.

[1431] "Analysis Center" means a facility or system that processes and analyzes received data.

[1432] "Generative AI" is an AI that has the ability to learn large amounts of data and analyze and generate data that is given to it.

[1433] "Traffic volume" is an indicator that indicates the number of vehicles or people passing through a specific location within a certain period of time.

[1434] "Vehicle type" is a classification used to distinguish between different types of automobiles.

[1435] "Attribute information of pedestrians" is information including characteristics of pedestrians such as gender, age, and clothing.

[1436] A "database" is a system for efficiently storing, searching, and managing data.

[1437] "Report" means a report or summary of information generated based on analyzed data.

[1438] "User-friendly" is a term that describes something that is easy for users to use and understand.

[1439] A "sensor" is a device that detects a physical quantity (e.g., temperature, humidity, movement, etc.) and converts it into data.

[1440] "Emotion recognition data" is data used to identify emotions from human facial expressions, voice, movements, etc.

[1441] "Display adaptation" refers to changing how information is displayed in response to user factors.

[1442] "Guidance" is a term that means providing guidance to a destination or information.

[1443] The system that specifically embodies this invention is composed of an unmanned aerial vehicle (HAPS), a server, an analysis center, a database, and terminals and sensors within the autonomous vehicle.

[1444] The server first receives video data acquired from an unmanned aerial vehicle (HAPS). The unmanned aerial vehicle is capable of capturing images of a wide area and transmitting real-time video data. The received video data is temporarily stored by the server and then sent to the analysis center.

[1445] The analysis center uses generative artificial intelligence to analyze the video data. Specifically, information on traffic volume, vehicle types, and pedestrian attributes is obtained. Examples of generative artificial intelligence used include OpenAI's GPT and Google's DeepMind.

[1446] The analyzed data is sent to a server, which categorizes it and stores it in a database. The database can be MySQL or PostgreSQL. The server then automatically generates and stores reports based on the categorized data.

[1447] The devices inside the autonomous vehicles are equipped with cameras and microphones that detect the emotions of the driver and passengers. Emotion recognition engines such as Microsoft Azure Cognitive Services and Amazon Rekognition are used for emotion recognition. Emotion data is sent to a server, which then adjusts the display and guidance based on this data.

[1448] As a concrete example, while an autonomous vehicle is driving in an urban area, HAPS captures real-time images of the surrounding traffic conditions. This image data is sent to a server and then analyzed as traffic information at an analysis center. The captured traffic information is then sent to the server and stored in a database. If the driver is feeling stressed, the sensor sends this emotional data to the server, which then generates a report suggesting the optimal detour route. The report is then displayed on the device, providing the driver with comfortable navigation.

[1449] An example prompt is:

[1450] "Explain the system that analyzes video data from HAPS, obtains traffic information, and provides the optimal navigation route based on the driver's emotional data."

[1451] This will enable the provision of real-time traffic information that responds to the user's emotions and safe and comfortable navigation.

[1452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1453] Step 1:

[1454] The server receives video data from the unmanned aerial vehicle. The HAPS captures a wide area and transmits real-time video data. This video data is input to the server and temporarily stored. The input data is stored in temporary storage and then goes to the next analysis step.

[1455] Step 2:

[1456] The server sends the received video data to the analysis center. Using a pre-configured communication protocol, the server transfers the stored video data to the analysis center. The input data is the video data from HAPS, and the output data is the video data transferred to the analysis center.

[1457] Step 3:

[1458] The analysis center uses generative artificial intelligence to analyze the video data and obtain information on traffic volume, vehicle types, and pedestrian attributes. The input data is the video data sent from HAPS, and a generative AI model (e.g., OpenAI's GPT or Google's DeepMind) is used. The analyzed data is extracted, and traffic information is obtained as output data. The obtained information includes the number of vehicles, vehicle type identification, and pedestrian attributes (gender, age, etc.).

[1459] Step 4:

[1460] The server receives traffic information sent from the analysis center, categorizes it into categories, and stores it in a database. The input data is the traffic information sent from the analysis center, and the output data is database entries organized by category. The server does this using a database management system (e.g., MySQL, PostgreSQL).

[1461] Step 5:

[1462] The server automatically generates and saves reports based on the classified data. The generated reports include information such as increases or decreases in traffic volume during specific time periods, vehicle distribution, and pedestrian attributes. The input data is the categorized traffic information stored in the database, and the output data is the generated report.

[1463] Step 6:

[1464] The terminal inside the autonomous vehicle displays the report and recognizes the user's emotions using sensors inside the vehicle. An emotion recognition engine (e.g., Microsoft Azure Cognitive Services, Amazon Rekognition) is used, and the input data is emotion information obtained from the camera and microphone. The output data is the analyzed emotion data.

[1465] Step 7:

[1466] The server adjusts the display and guidance based on emotion recognition data. The input data is emotion data sent from the emotion recognition engine, and the output data is adjusted navigation and information display. For example, if the driver is feeling stressed, the server will optimize the route and suggest a more comfortable route.

[1467] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1468] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1469] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1470] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1471] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1472] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1473] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1474] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1475] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1476] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1477] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1478] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1479] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1480] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1481] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1482] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1483] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1484] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1485] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1486] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1487] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1488] The following is further disclosed regarding the above embodiment.

[1489] (Claim 1)

[1490] means for receiving video data from the unmanned aerial vehicle;

[1491] means for transmitting the received video data to an analysis center;

[1492] A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes;

[1493] a means for categorizing the obtained information and storing it in a database;

[1494] A means for generating and storing reports based on the classified data;

[1495] A system including a means for providing generated reports in a user-friendly format.

[1496] (Claim 2)

[1497] 10. The system of claim 1,

[1498] The system includes a means for analyzing the acquired data in real time and automatically generating reports.

[1499] (Claim 3)

[1500] 10. The system of claim 1,

[1501] A system that receives video data from an unmanned aerial vehicle over a wide area and includes a means for analyzing the detailed visual data.

[1502] "Example 1"

[1503] (Claim 1)

[1504] means for receiving video data from the unmanned aerial vehicle;

[1505] means for transmitting the received video data to an analysis center;

[1506] A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes;

[1507] a means for categorizing the obtained information and storing it in a database;

[1508] A means for generating and storing reports based on the classified data;

[1509] A means of presenting the generated reports in a user-friendly format;

[1510] means for converting the format of the received video data;

[1511] A means for receiving analysis results from the analysis center and classifying them into each data category;

[1512] a means for indexing data stored in the database;

[1513] means for generating a report based on a template;

[1514] A system that includes a means to access reports from a terminal and display them at a specific date, time, or location.

[1515] (Claim 2)

[1516] 10. The system of claim 1, further comprising means for analyzing the acquired data in real time and automatically generating a report.

[1517] (Claim 3)

[1518] 10. The system of claim 1, including means for receiving wide-area video data from the unmanned aerial vehicle and analyzing the detailed visual data.

[1519] "Application Example 1"

[1520] (Claim 1)

[1521] means for receiving video data from the unmanned aerial vehicle;

[1522] means for transmitting the received video data to an analysis center;

[1523] A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes;

[1524] a means for categorizing the obtained information and storing it in a database;

[1525] A means for generating and storing reports based on the classified data;

[1526] A means of presenting the generated reports in a user-friendly format;

[1527] A system that includes a means for generating and providing information to optimize route selection and operation management of automated vehicles.

[1528] (Claim 2)

[1529] 10. The system of claim 1, further comprising means for analyzing the acquired data in real time and automatically generating a report.

[1530] (Claim 3)

[1531] 10. The system of claim 1, including means for receiving wide-area video data from the unmanned aerial vehicle and analyzing the detailed visual data.

[1532] "Example 2: Combining Emotion Engines"

[1533] (Claim 1)

[1534] means for receiving video data from the unmanned aerial vehicle;

[1535] means for transmitting the received video data to an analysis center;

[1536] A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes;

[1537] a means for categorizing the obtained information and storing it in a database;

[1538] A means for generating and storing reports based on the classified data;

[1539] A means of presenting the generated reports in a user-friendly format;

[1540] A means for the device to recognize the user's emotions and collect emotion data;

[1541] A system including means for adjusting the display of a report based on emotion data.

[1542] (Claim 2)

[1543] 10. The system of claim 1, further comprising means for analyzing the acquired data in real time and automatically generating a report.

[1544] (Claim 3)

[1545] 2. The system according to claim 1, wherein the terminal includes means for recognizing the user's emotions and analyzing the emotion data to reflect the feedback in the report.

[1546] "Application example 2 when combining emotion engines"

[1547] (Claim 1)

[1548] means for receiving video data from the unmanned aerial vehicle;

[1549] means for transmitting the received video data to an analysis center;

[1550] A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes;

[1551] a means for categorizing the obtained information and storing it in a database;

[1552] A means for generating and storing reports based on the classified data;

[1553] A means of presenting the generated reports in a user-friendly format;

[1554] a means for recognizing a user's emotion using sensors in the vehicle;

[1555] A system including means for adjusting display or guidance based on emotion recognition data.

[1556] (Claim 2)

[1557] 10. The system of claim 1, wherein the acquired data is analyzed in real time and reports and navigation are automatically adjusted based on the emotion recognition data.

[1558] (Claim 3)

[1559] 10. The system of claim 1, which receives wide-area video data from unmanned aerial vehicles, analyzes the detailed visual data, and uses a generative AI model to adapt to driver emotional data in real time. [Explanation of symbols]

[1560] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving video data from the unmanned aerial vehicle; means for transmitting the received video data to an analysis center; A means to analyze the video data using generative artificial intelligence at the analysis center and obtain information on traffic volume, vehicle types, and pedestrian attributes; a means for categorizing the obtained information and storing it in a database; A means for generating and storing reports based on the classified data; A system including a means for providing generated reports in a user-friendly format.

2. 10. The system of claim 1, The system includes a means for analyzing the acquired data in real time and automatically generating reports.

3. 10. The system of claim 1, A system that receives video data from an unmanned aerial vehicle over a wide area and includes a means for analyzing the detailed visual data.

Images