System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving image data and sensor data from an aerial vehicle; means for executing an image recognition algorithm to identify anomalies in infrastructure based on the received data; means for automatically generating an inspection report based on the identified anomalies; and means for notifying an administrator of the generated inspection report.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As infrastructure continues to deteriorate, society today demands appropriate inspections and prompt repairs. However, carrying out inspection and repair work manually poses problems due to the enormous workload and labor shortages. Furthermore, many inspection points are located at high altitudes or in places that are difficult to access, making safety a major issue. Furthermore, efficient analysis of inspection results and the rapid development of repair plans require advanced technology and specialized knowledge, making it essential to improve work efficiency. [Means for solving the problem]

[0005] This system uses an aerial drone to collect image data and sensor data from infrastructure structures and transmits that data to a server. The server then runs an image recognition algorithm based on the received data to identify abnormalities such as cracks, rust, and corrosion. The server then automatically generates an inspection report based on the identified abnormalities and notifies the administrator. It also analyzes sensor data to evaluate seismic intensity, temperature, humidity, pressure, and other factors to clarify the type and severity of the abnormality. This system enables infrastructure inspection and repair to be carried out efficiently and safely with a small number of people, resolving issues caused by personnel and technical shortages.

[0006] An "aerial vehicle" is an unmanned device that flies through the sky and collects image data and sensor data.

[0007] "Image data" is digital data containing visual information of infrastructure structures photographed by an aerial vehicle.

[0008] "Sensor data" is data that is measured and recorded in digital format for physical conditions such as temperature, humidity, pressure, and seismic intensity.

[0009] "Infrastructure" refers to public and industrial infrastructure such as roads, bridges, tunnels, water and sewerage systems, and power lines.

[0010] An "image recognition algorithm" is a computer program that analyzes image data and detects specific patterns (e.g., cracks, rust, corrosion).

[0011] An "inspection report" is a document that describes the condition of an infrastructure structure, any abnormalities, and the evaluation results based on analyzed data.

[0012] A "server" is a computer system that receives, processes, and stores data.

[0013] "Administrator" refers to the person in charge or responsible for monitoring the inspection and repair work of infrastructure and taking any necessary action.

[0014] A "repair plan" is a plan that describes the repair schedule and procedures that are drawn up based on the inspection results.

[0015] "Notification" refers to the act of informing a person in charge of important information such as analysis results and inspection reports using electronic means. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports, reducing the burden on engineers and enabling efficient and safe infrastructure inspections.

[0038] System Configuration

[0039] The system consists of the following main components:

[0040] 1. Aerial Vehicles

[0041] The drones collect images and sensor data of infrastructure structures as they fly.

[0042] 2. Server

[0043] The server receives, stores, and analyzes the data sent from the drone.

[0044] 3. User Interface Terminal

[0045] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0046] Program processing explanation

[0047] Data collection

[0048] The user uses a dedicated terminal to set the location information of the infrastructure to be inspected and a flight plan for the drone. Once the settings are complete, the terminal sends these instructions to the server. The server issues instructions to the drone based on the received information, causing the drone to fly automatically along the specified flight route. During flight, the drone uses cameras and various sensors to acquire image data and sensor data of the infrastructure, and sends this to the server in real time.

[0049] Data analysis

[0050] The server runs an image recognition algorithm on the received image data to identify abnormalities such as cracks, rust, and corrosion. The image recognition algorithm uses pre-trained pattern recognition technology to detect abnormalities with high accuracy. It also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity. This allows it to pinpoint abnormalities in detail.

[0051] Reporting and Notifications

[0052] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this inspection report in PDF format and emails it to managers and other relevant parties. It also has an alert function that ensures immediate notification if any important anomalies are discovered.

[0053] User Interface

[0054] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays the location of any abnormalities and detailed analysis results, and past inspection data can also be easily searched and referenced. This enables quick and efficient response.

[0055] Specific examples

[0056] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can then check the report on their device and develop a repair plan as needed.

[0057] This will enable efficient measures to address the deterioration of infrastructure and resolve problems caused by labor shortages.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The user uses a dedicated terminal to specify the infrastructure to be inspected and set up a flight plan, which includes the location of the inspection site, the drone's flight path, and the time of day.

[0061] Step 2:

[0062] The terminal sends the set flight plan to the server, which receives it and sends flight plan instructions to the drone.

[0063] Step 3:

[0064] The drone automatically flies a predetermined route according to a flight plan, and during flight, it uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure.

[0065] Step 4:

[0066] The drones transmit the collected image data and sensor data in real time to a server, which receives the data and stores it in a database.

[0067] Step 5:

[0068] The server then runs image recognition algorithms on the received image data, which automatically detects abnormalities such as cracks, rust, and corrosion.

[0069] Step 6:

[0070] The server performs a detailed analysis of the detected anomalies, identifying their location, range, depth, etc., and generates the analysis results as text data.

[0071] Step 7:

[0072] The server also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, pressure, etc. If an abnormality is detected, it also identifies the location, type, and severity of the abnormality.

[0073] Step 8:

[0074] The server automatically generates an inspection report based on the analysis results, which includes detailed information on identified abnormalities and the priority of areas requiring repair.

[0075] Step 9:

[0076] The server outputs the generated inspection report in PDF format and notifies the administrator and other relevant parties by email. In addition, if any abnormalities are found to be urgent, an alert will be sent immediately.

[0077] Step 10:

[0078] Users can check inspection results in real time through a dashboard on a dedicated device, which displays details of abnormalities and analysis results, and can also easily refer to past inspection data.

[0079] Step 11:

[0080] Users can create repair plans via their terminals, which include the necessary work and schedules.

[0081] Step 12:

[0082] The server stores the details of the repair plan and updates the database so that it can be used at the next inspection.

[0083] This will enable the creation of a system that allows for efficient and safe infrastructure inspection and repair.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Modern infrastructure structures are diverse, and their inspections require a huge amount of time and effort. Furthermore, traditional inspection methods require engineers to work at height or in dangerous environments, threatening their safety. Furthermore, manual inspections carry a high risk of overlooking or false positives, creating a need for efficient and highly accurate inspections.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for transmitting location information and a flight plan of the inspection target from a user terminal, means for executing an image recognition algorithm for identifying abnormalities in the infrastructure based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, and means for displaying the analysis results of the abnormalities to the user in real time. This enables efficient and high-precision inspection of infrastructure, reduces the burden on engineers, and improves safety.

[0089] An "aerial vehicle" is an unmanned flying device, such as a drone. It is primarily used to collect image data and sensor data over a wide area.

[0090] "Image data" refers to photographic images and videos taken by aerial vehicles, and is information used to visually grasp the appearance and condition of infrastructure structures.

[0091] "Sensor data" refers to physical data collected from various sensors (e.g., temperature sensors, humidity sensors, seismic intensity sensors) mounted on airborne vehicles, and is information used to numerically grasp environmental conditions and the status of infrastructure structures.

[0092] "User terminal" refers to an electronic device operated by a user, such as a tablet or laptop computer with dedicated software installed. It is used to set and transmit inspection target location information and flight plans.

[0093] A "server" is a computer that stores and processes data on a network, and receives, analyzes, stores, and reports data sent from airborne vehicles.

[0094] An "image recognition algorithm" is a technology for analyzing image data to detect specific patterns or abnormalities, and includes pattern recognition technology that uses AI and machine learning.

[0095] An "inspection report" is a document that summarizes the inspection results, including any identified abnormalities and the priority of necessary repair work. It is used to report to managers and other relevant parties.

[0096] "Real-time display" means that data is displayed to the user as soon as it is collected, allowing for rapid response in the field.

[0097] "Abnormal areas" refer to problems such as cracks, rust, and corrosion that exist in infrastructure structures, and are areas that require repair or restoration.

[0098] This invention is a system that uses an aerial vehicle to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. This system reduces the burden on engineers and enables efficient and safe infrastructure inspections.

[0099] System Configuration

[0100] The system consists of the following main components:

[0101] 1. Aerial Vehicles

[0102] Drones such as the DJI Phantom 4 and DJI Mavic 2 fly over the infrastructure and collect images and sensor data.

[0103] 2. Server

[0104] The server receives, stores, and analyzes the data. For example, a cloud computing service such as AWS (registered trademark) EC2 or Google (registered trademark) Cloud is used.

[0105] 3. User Interface Terminal

[0106] A terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports. Tablets and laptops are commonly used as terminals.

[0107] Data collection

[0108] The user uses a dedicated device to input the location information of the infrastructure to be inspected and the flight plan into the drone. This is done using a tablet or laptop with a dedicated application installed. Once the settings are complete, the device sends these instructions to a server via Wi-Fi or 4G / 5G networks.

[0109] The server issues instructions to the drone based on the received information, causing it to fly automatically along the designated route. The drone is equipped with a camera and various sensors, and acquires image data and sensor data (temperature, humidity, seismic intensity, etc.) of infrastructure structures during flight, which is then sent to the server in real time.

[0110] Data analysis

[0111] When the server receives the transmitted image data, it analyzes the data using image recognition algorithms such as TENSORFLOW (registered trademark) and OpenCV. The analysis algorithm uses pre-trained pattern recognition rules to identify abnormalities such as cracks, rust, and corrosion. At the same time, it also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, and seismic intensity. This makes it possible to pinpoint the location of any abnormalities in detail.

[0112] Reporting and Notifications

[0113] Once the analysis is complete, the server automatically generates an inspection report using Python and JavaScript (registered trademark). The generated report contains detailed information about any identified abnormalities and the priority of any necessary repair work. The server generates this report in PDF format and sends it via email to administrators and other relevant parties using the SMTP protocol. In addition, if a serious abnormality is discovered, an alert notification can be sent immediately using the Twilio API, etc.

[0114] User Interface

[0115] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays abnormalities and detailed analysis results. This uses an interactive user interface built using React and Angular.js. Users can also easily search and reference past inspection data, enabling quick and efficient response.

[0116] Specific examples

[0117] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server analyzes the received data in real time and identifies cracks and corrosion. An inspection report is automatically generated based on the detection results and sent to the user's device, where the user can review the report and develop any necessary repair plans.

[0118] An example of a prompt sentence to be input to a specific generative AI model:

[0119] Prepare a bridge inspection report using the following information:

[0120] Name of the bridge: XX Bridge

[0121] Location information:35.710063, 139.8107

[0122] Acquired data: Image data (24 images in total), sensor data (temperature, humidity, seismic intensity)

[0123] Abnormalities: Cracks (5 places), rust (3 places)

[0124] Details of the abnormality: average crack length 0.5m, average rust area 10cm²

[0125] Based on this, generate an inspection report in PDF format.

[0126] The present invention makes it possible to efficiently address the deterioration of infrastructure, thereby resolving labor shortages and safety issues.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1: Setting up input data

[0129] The user uses a dedicated device to set the location information and flight plan for the infrastructure to be inspected. Input is made using a tablet or laptop, and detailed local coordinate information (latitude and longitude) and flight route are entered through a dedicated application. This generates instructions for the drone to navigate along the flight path.

[0130] Step 2: Submit your flight plan

[0131] The device transmits the location information of the inspection target and the flight plan set by the user to a server via Wi-Fi or 4G / 5G network. The server analyzes the received data and determines the route the drone should take. The generated data is then passed to the drone control API.

[0132] Step 3: Start flying the drone automatically

[0133] The server issues automatic flight instructions to the drone, using the drone's built-in flight control system. As the drone flies automatically along a set route, it uses its camera to capture image data of infrastructure structures and its sensors to collect physical data such as temperature, humidity, and seismic intensity. The collected data is sent to the server in real time for subsequent data analysis processing.

[0134] Step 4: Receiving and storing data

[0135] The server receives the image data and sensor data sent from the drone and first stores it in a database. The received data includes large amounts of image files and sensor output, which are later used for analytical processing. Amazon S3 or Google Cloud Storage are often used for server storage.

[0136] Step 5: Analyze imagery and sensor data

[0137] The server runs image recognition algorithms on the received and stored image data. For example, it uses TensorFlow or OpenCV to identify abnormalities such as cracks, rust, and corrosion. It analyzes the input data (image data) for image recognition processing and generates output results indicating abnormalities. These results are saved in JSON format and used to generate subsequent reports. At the same time, it also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity.

[0138] Step 6: Generate an inspection report

[0139] The server automatically generates a draft inspection report based on the results of the image and sensor data analysis. It generates the report in PDF format using Python's ReportLab library, among other tools. The report includes detailed information about identified anomalies and the priority of recommended repair work. The report is saved for future notification.

[0140] Step 7: Report Notifications and Alerts

[0141] The server notifies the administrator and relevant parties of the generated inspection report. The report is sent by email using the SMTP protocol. If a serious abnormality is detected, an alert is sent immediately via SMS or phone using the Twilio API. This makes it possible to respond quickly when a rapid response is required.

[0142] Step 8: Viewing in the User Interface

[0143] Users can view inspection results in real time via a dashboard on a dedicated device. The device dashboard provides an interactive interface built using React and Angular.js. Details of abnormalities and analysis results are visually displayed here, and past inspection data can be easily searched and referenced. This enables quick and efficient response.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Inspecting equipment within a factory requires inspecting high places and complex structural parts, and conventional methods place a heavy burden on personnel and pose safety issues. Furthermore, identifying abnormalities and creating inspection reports was done manually, which was inefficient and prone to errors. For this reason, there was a need for a system that could inspect factory equipment safely and efficiently and identify abnormalities in real time.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for receiving image data and sensor data from equipment in the factory, means for executing an image recognition algorithm using AI technology to identify abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities and saving the generated report in a cloud server, and means for notifying an administrator of the generated inspection report in real time. This enables safe and efficient inspection of factory equipment, enables abnormalities to be identified in real time, and enables rapid response.

[0149] "Factory facilities" refers to the infrastructure used in factories, such as various machines, devices, piping, and electrical equipment.

[0150] "Image data" is data representing visual information captured using a camera or other imaging device.

[0151] "Sensor data" refers to data representing physical or environmental information obtained from various sensors.

[0152] "Means for receiving" refers to a combination of hardware and software for remotely obtaining data and processing the obtained data.

[0153] An "image recognition algorithm" is a computational procedure for analyzing image data and detecting specific patterns or anomalies within it.

[0154] "AI technology" is a technology that uses artificial intelligence to analyze data and perform pattern recognition.

[0155] "Automatic generation" refers to the process where a system automatically generates a report based on data without human intervention.

[0156] A "cloud server" is a remote virtual server used to store and process data over the Internet.

[0157] "Means of real-time notification" refers to a method or device that immediately transmits information to managers and relevant parties without delay when an abnormality is detected.

[0158] In this invention, we build a system for efficiently and safely inspecting equipment in factories. The system consists of a drone, a cloud server, and a smartphone app. To implement this invention, the following hardware and software are used.

[0159] Hardware

[0160] Drone: Equipped with a camera, temperature and humidity sensors, seismometer, and pressure sensor.

[0161] Cloud server: A remote server (typically AWS or Google Cloud) for storing and analyzing data.

[0162] User device: smartphone or tablet (iOS or ANDROID (registered trademark) based).

[0163] software

[0164] Drone control software: Used for automated drone flight and data collection (e.g., DJI SDK).

[0165] Server communication library: Sends and receives data from the drone to the cloud server (e.g., HTTP communication library).

[0166] AI analysis library: Analyzes image data and sensor data to identify abnormalities (TensorFlow, OpenCV, etc.).

[0167] Report generation library: Automatically generate inspection reports (e.g., ReportLab).

[0168] Notification system: Provides real-time notifications to user devices (e.g., Firebase Cloud Messaging).

[0169] User interface: The application that runs on a smartphone or tablet.

[0170] System Operation

[0171] 1. Data Collection

[0172] Users use a smartphone app to enter information about the equipment they want to inspect and their inspection plan. Once the settings are complete, the cloud server transmits the information to the drone. The drone then flies automatically and uses its camera and sensors to collect image data and sensor data from the equipment. The data is then sent to the cloud server in real time.

[0173] 2. Data Analysis

[0174] The cloud server then runs an image recognition algorithm using AI technology on the received data. This algorithm detects abnormalities such as cracks, rust, and corrosion, and also identifies abnormalities in high places and complex structural parts. The results of this analysis are then stored on the cloud server.

[0175] 3. Report generation and notifications

[0176] Based on the analysis results, the cloud server automatically generates an inspection report and stores it in the cloud. If any significant abnormalities are detected, a notification system sends an alert to the user's device in real time. Users can check the inspection results on their smartphones or tablets and take any necessary measures.

[0177] Specific examples

[0178] For example, when conducting regular inspections of pipelines within a factory, the user launches the app and inputs the location information of the pipeline to be inspected and the inspection plan. The drone flies automatically according to instructions, capturing sensor data while photographing the area around the pipeline. The cloud server receives this data and performs AI analysis. If cracks, rust, etc. are identified, an inspection report is immediately generated and a notification is sent to the user's device. The user can check the report on the app and create a maintenance plan.

[0179] Prompt Sentence Examples

[0180] "We are thinking about a system that uses drones to detect abnormalities in factory equipment. In this system, drones fly automatically to collect image data and sensor data from the equipment, which is then analyzed by AI to identify any abnormalities. Specifically, please explain in detail what steps should be taken to build the system, including the hardware and software to be used."

[0181] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0182] Step 1:

[0183] The user launches the smartphone app and inputs information about the equipment they want to inspect (for example, the equipment's location and inspection plan). This information is used as input. The smartphone app then sends this input data to a cloud server. The data sent to the server becomes the output for creating an automatic flight plan for the drone.

[0184] Step 2:

[0185] The cloud server issues automatic flight instructions to the drone based on the received inspection plan. Once the flight plan is sent to the drone, the drone automatically flies along the designated route, collecting image data using its camera and sensor data using various sensors (temperature and humidity sensors, seismometers, pressure sensors). This becomes input data. The collected image data and sensor data are sent to the cloud server in real time and become output for analysis.

[0186] Step 3:

[0187] The cloud server runs an image recognition algorithm using AI technology on the image data and sensor data it receives in real time. Image data and sensor data are used as input data. The cloud server analyzes the image data using AI technology (for example, TensorFlow or OpenCV) to detect abnormalities such as cracks, rust, and corrosion. Detailed data on detected abnormalities is output.

[0188] Step 4:

[0189] The cloud server automatically generates an inspection report based on details of abnormalities detected by the image recognition algorithm. Detailed data on the abnormalities is used as input data. The server converts the generated report into PDF format and saves it in the cloud. The generated inspection report is the output.

[0190] Step 5:

[0191] The cloud server prepares the generated inspection report for notification to the administrator. The generated inspection report is used as input data. The server notifies the administrator via a notification system (e.g., Firebase Cloud Messaging) that the report is available to their smartphone or tablet. This is the output. If any significant anomalies are detected, an alert is sent immediately.

[0192] Step 6:

[0193] Users can check the inspection report through the application on their smartphone or tablet. The inspection report received after receiving the notification is used as input data. Users can check detailed abnormalities and inspection results through the app interface and plan necessary countermeasures. The checked report contents are output.

[0194] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0195] This invention is a system that uses an aerial vehicle (drone) to collect and analyze image data and sensor data of infrastructure structures, generates inspection reports, and combines this with an emotion engine that recognizes the user's emotions. This system reduces the burden on engineers, enables efficient and safe infrastructure inspections, and presents information that takes user emotions into consideration.

[0196] System Configuration

[0197] The system consists of the following main components:

[0198] 1. Aerial Vehicles

[0199] The drones collect images and sensor data of infrastructure structures as they fly.

[0200] 2. Server

[0201] The server receives, stores, and analyzes the data sent from the drone.

[0202] 3. User Interface Terminal

[0203] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0204] 4. Emotion Engine

[0205] The emotion engine recognizes the user's emotions by analyzing their facial expressions and voice, and then displays appropriate information and notifications accordingly.

[0206] Program processing explanation

[0207] Data collection

[0208] The user uses a dedicated terminal to specify the infrastructure inspection target and set up a flight plan. The settings include the location information of the inspection location, the drone's flight path, and the time period. The terminal sends the set flight plan to the server. The server receives this and sends flight plan instructions to the drone. The drone follows the instructed flight plan and automatically flies the predetermined flight path. During flight, the drone uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure. The drone sends the collected image data and sensor data to the server in real time.

[0209] Data analysis

[0210] The server runs an image recognition algorithm on the received image data. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities, identifying the location, range, depth, etc. of the abnormality and generating the analysis results as text data. It also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, and pressure. If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0211] Report Generation

[0212] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and other relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0213] User Interface and Emotion Engine

[0214] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0215] Specific examples

[0216] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0217] This will enable efficient measures to deal with aging infrastructure and enable system operations that take user feelings into consideration.

[0218] The processing flow will be explained below.

[0219] Step 1:

[0220] Using a dedicated terminal, the user selects the infrastructure to be inspected and sets up a flight plan, including details such as the target location, flight path, shooting interval, type of sensor to be used, and flight time.

[0221] Step 2:

[0222] The terminal sends the configured flight plan to the server, which receives the flight plan and transmits it to the drone.

[0223] Step 3:

[0224] The drone will automatically begin flying according to the received flight plan, and during the flight, it will use its cameras and sensors to collect image and sensor data.

[0225] Step 4:

[0226] The drone transmits the collected image data and sensor data to a server in real time.

[0227] Step 5:

[0228] The server runs an image recognition algorithm on the received image data, which uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[0229] Step 6:

[0230] The server identifies any abnormalities detected through image analysis and converts their location, range, and depth into text data as the analysis results.

[0231] Step 7:

[0232] The server also analyzes sensor data, such as seismic intensity, temperature, humidity, and pressure, and if an abnormality is detected, identifies the type and severity of the abnormality.

[0233] Step 8:

[0234] The server automatically generates an inspection report based on the analysis results, which includes text and diagrams and includes detailed information about any abnormalities and the priority of repairs.

[0235] Step 9:

[0236] The server notifies the administrator of the generated inspection report via email or SMS, with the report itself attached in PDF format. It also sends an immediate alert for any abnormalities that are deemed urgent.

[0237] Step 10:

[0238] Users can check the inspection and analysis results in real time through a dashboard on a dedicated device, which also displays detailed information on abnormalities and past inspection data.

[0239] Step 11:

[0240] The emotion engine recognizes the user's emotional state as they use the dashboard. The emotion engine analyzes the user's facial expressions and voice in real time to identify the user's emotional state.

[0241] Step 12:

[0242] The device adjusts the content and display of notification messages received by the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the device will display a notification message that gives a sense of security.

[0243] Step 13:

[0244] Users can review coordinated notifications and inspection reports and create necessary repair plans, which include specific work content and schedules.

[0245] Step 14:

[0246] The server stores the proposed repair plan in a database and updates the data so that it can be used at the next inspection.

[0247] This will enable efficient and safe infrastructure inspection and repair, while also providing information that takes user feelings into consideration.

[0248] Example 2

[0249] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0250] In conventional infrastructure inspection systems, the analysis and reporting of inspection results was done manually, which took time and effort and could lead to variations in the accuracy of the analysis. This also placed a heavy burden on engineers, posing safety and efficiency challenges. Furthermore, the system did not present information that took user emotions into consideration, potentially increasing user anxiety and stress.

[0251] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0252] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, emotion analysis means for recognizing the user's emotional state and adjusting the notification message, and user interface means for displaying data in real time. This enables infrastructure inspection work to be performed efficiently and safely, reducing the burden on engineers. Furthermore, since information is presented taking the user's emotions into consideration, it is possible to reduce user anxiety and stress.

[0253] "Aerial vehicle" is a general term for unmanned aerial vehicles that fly over infrastructure structures such as stores and facilities to collect data.

[0254] "Image data" refers to visual information of infrastructure structures photographed by a camera mounted on an aerial vehicle.

[0255] "Sensor data" refers to physical property data such as temperature, humidity, pressure, and seismic intensity measured by various sensors mounted on airborne vehicles.

[0256] "Means for receiving" means the equipment or software functions for receiving and storing image data and sensor data transmitted from an airborne vehicle.

[0257] An "image recognition algorithm" is a program or method for automatically identifying abnormalities such as cracks, rust, and corrosion in received image data.

[0258] An "inspection report" is an automatically generated report based on abnormalities identified by an image recognition algorithm, and includes detailed information about the abnormalities and the priority of repair work.

[0259] The "means for notifying the administrator" refers to a communication means or software function for notifying the administrator or related parties of the generated inspection report in real time.

[0260] The "emotion analysis means" is a device or software function for analyzing the user's facial expressions and voice and recognizing the user's emotions.

[0261] "User interface means" refers to the function of a device or software that provides an interface that allows users to view and manipulate inspection data and results in real time.

[0262] This invention relates to a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. Furthermore, by combining this with an emotion engine that recognizes user emotions and presents appropriate information, it is possible to reduce the burden on engineers and perform infrastructure inspections efficiently and safely.

[0263] To implement this system, the following major hardware and software are used:

[0264] Hardware

[0265] 1. Aerial vehicle (drone): Equipped with a camera and multiple sensors (temperature and humidity sensors, pressure sensors, etc.), it has the ability to collect image data and sensor data.

[0266] 2. Server: A high-performance computing system for receiving, storing, and analyzing data sent from the drones.

[0267] 3. User interface terminal: A device operated by the user that provides an interface for issuing inspection instructions and checking analysis results and inspection reports.

[0268] software

[0269] 1. Image recognition algorithm: An image analysis program for detecting abnormalities such as cracks, rust, and corrosion. It is composed mainly of deep learning models.

[0270] 2. Data analysis software: A program that analyzes sensor data (temperature, humidity, pressure, seismic intensity, etc.) and evaluates physical properties.

[0271] 3. Emotion analysis engine: Software that analyzes the user's facial expressions and voice to recognize their emotional state.

[0272] 4. User interface software: An interface program designed to display data in real time and allow users to easily operate it.

[0273] Specific actions

[0274] The user uses a dedicated device to specify the infrastructure to be inspected and set up a flight plan, including the location of the inspection site, the drone's flight path, and the time of day. The flight path includes detailed instructions, such as covering both the top and bottom of a bridge.

[0275] The terminal sends the set flight plan to the server. The server receives it and sends flight plan instructions to the drone. The drone follows the instructed flight plan, automatically flying along a predetermined flight path while using cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure, and sends this data to the server in real time.

[0276] The server stores the received data in a database and runs an image recognition algorithm on the image data. This algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities to identify their location, range, depth, etc. It also analyzes sensor data to evaluate physical characteristics (temperature, humidity, seismic intensity, pressure, etc.). If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0277] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0278] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0279] Specific examples

[0280] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0281] Prompt Sentence Examples

[0282] 1. Drone flight plan configuration: "Please explain the process for configuring infrastructure location information and inspection plans (flight path, time slots, etc.) and sending them to the server."

[0283] 2. Data Analysis: Please explain in detail how the server analyzes the received image data and sensor data to detect any abnormalities.

[0284] 3. Report Generation: "Please explain in detail the steps by which the server automatically generates an inspection report based on the analysis results."

[0285] 4. How the Emotion Engine Works: "Please explain how the emotion engine works, recognizing the user's emotions and displaying appropriate information or notifications."

[0286] This will enable efficient measures to deal with aging infrastructure, and will also enable system operations that take user feelings into consideration.

[0287] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0288] Step 1:

[0289] The user uses a dedicated terminal to specify the infrastructure structure (e.g., a bridge) to be inspected. Input includes the location information of the inspection location, the drone's flight path, and the flight time. The terminal compiles this information into a drone flight plan and generates data in JSON format. This data becomes the terminal's output.

[0290] Step 2:

[0291] The device sends the generated flight plan data to the server. The server receives this data and generates instruction data for the drone based on it. The instruction data includes the specific flight path, subject to be photographed, and timing of the photograph. The server's output is instruction data for the drone.

[0292] Step 3:

[0293] The server sends instruction data to the drone. Based on the received instruction data, the drone automatically begins flying according to the set flight path. During flight, the drone uses its on-board camera and sensors to collect image data and sensor data. This is the drone's output data.

[0294] Step 4:

[0295] The drone transmits the collected image data and sensor data to the server in real time. The server receives this data and stores it in a database. The stored image data and sensor data become the input for the server.

[0296] Step 5:

[0297] The server receives the stored image data as input and runs an image recognition algorithm. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. Information about the detected abnormalities (location, range, depth, etc.) is output from the server. The server also analyzes sensor data and evaluates physical properties (temperature, humidity, seismic intensity, pressure, etc.). The results of these analyses are also output from the server.

[0298] Step 6:

[0299] The server automatically generates an inspection report based on the results of the image recognition algorithm and sensor data analysis. The report contains detailed information about any identified abnormalities and the priority of repair work. The generated inspection report is the output of the server.

[0300] Step 7:

[0301] The server outputs the generated inspection report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. This is the output of the server.

[0302] Step 8:

[0303] The generated inspection report is displayed on the dashboard of the user's dedicated terminal. Through the dashboard, the user can check the report contents and analysis results in real time. The dashboard display is the output of the user's terminal.

[0304] Step 9:

[0305] The emotion analysis engine analyzes the face and voice of the user using the dashboard to recognize the user's emotional state. The analysis results are the output of the emotion analysis engine. For example, if the user is feeling anxious, the engine will select and display a notification message that will give a sense of security. This is the final output on the user's device.

[0306] (Application example 2)

[0307] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0308] In modern logistics centers, inventory management and inspection work remains highly manual and difficult to perform efficiently. Furthermore, there is a lack of support that takes into consideration the emotions of operators, which can lead to excessive stress in the work. Infrastructure inspection work also places a heavy burden on technicians, and it must be performed efficiently and safely. To solve these issues, automation using drones and a system that recognizes users' emotions and provides appropriate feedback are needed.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0310] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying a manager of the generated inspection report, an emotion engine that recognizes a user's emotions, means for displaying information and notifying users based on their emotions, means for a drone to fly over a logistics center area and collect inventory data, and means for processing the collected inventory data and generating an inventory management report. This enables efficient inventory management at logistics centers, efficient inspections of infrastructure structures, and support that takes user emotions into consideration.

[0311] An "aerial vehicle" is an unmanned mechanical device capable of flying in the air, and in this context refers specifically to a drone.

[0312] "Image data" is data that includes visual information acquired by a photographic device such as a drone camera.

[0313] "Sensor data" refers to data about the physical characteristics of the environment and materials collected by various sensors mounted on drones.

[0314] An "image recognition algorithm" is a calculation method for extracting the characteristics of an object from image data and identifying abnormal areas.

[0315] An "inspection report" refers to a report generated based on the results of image recognition algorithms and sensor data analysis.

[0316] The "notification means" refers to a method or device for transmitting the generated inspection report or inventory control report to relevant parties or managers.

[0317] An "emotion engine" is a technology that analyzes a user's facial expressions and voice data to recognize their emotional state.

[0318] "Means for drones to fly over logistics center areas and collect inventory data" refers to a method for drones to fly autonomously over designated areas and obtain information on inventory items.

[0319] An "inventory management report" is a report that lists inventory status and abnormalities generated based on data collected by drones.

[0320] This invention is a system that uses aerial vehicles (drones) to collect and analyze data from infrastructure structures and logistics centers, improving the efficiency of inspections and inventory management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system displays and notifies information that takes the user's emotions into consideration. A specific embodiment of this system is described below.

[0321] Data collection

[0322] The server sends instructions to the drone based on a flight plan sent from the user interface terminal. The drone flies over designated infrastructure structures and logistics center areas, collecting image data using its camera and sensor data using various sensors. The collected data is sent to the server in real time.

[0323] Data analysis

[0324] The server uses image data sent from the drone to run image recognition algorithms and analyze abnormalities in infrastructure and inventory information within the logistics center. To detect abnormalities, it uses pattern recognition rules to identify cracks, rust, corrosion, and other abnormalities. It also analyzes sensor data and evaluates information such as temperature, humidity, seismic intensity, and pressure. Based on the results of this analysis, it automatically generates inspection reports for infrastructure and inventory management reports for the logistics center.

[0325] Reporting and Notifications

[0326] The server automatically generates a report of the analysis results, outputs the report in PDF format, and notifies administrators and other relevant parties by email. If a significant abnormality or inventory abnormality is discovered, an alert is sent immediately.

[0327] User Interface and Emotion Engine

[0328] Using a dedicated user interface terminal, users can specify inspection targets and inventory management areas and check the results in real time. The terminal displays detailed inspection results and inventory status, and past data can also be easily referenced. In addition, an emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling stressed, the server will select a reassuring notification message and display it to the user.

[0329] Specific examples

[0330] For example, when conducting an inventory inspection at a logistics center, the user sets the location information of inventory shelves and an inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the inventory shelves and collecting sensor data. The server receives this data in real time and uses image recognition algorithms to check inventory and detect abnormalities. Based on the detection results, the server automatically generates an inventory management report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately adjusted by the emotion engine.

[0331] Prompt Sentence Examples

[0332] "We will begin an inspection of inventory shelf area A. Please fly the drone into the area and scan the product barcodes."

[0333] "There is an abnormality in the data being analyzed. The upper shelf of inventory shelf A is misaligned and needs to be checked."

[0334] "Stress response detected. Providing advice for efficient stock replenishment."

[0335] This will enable efficient inventory management at logistics centers and inspection of infrastructure structures, as well as support that takes user emotions into consideration.

[0336] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0337] Step 1:

[0338] Users log in using a dedicated terminal and specify the inventory area of ​​the logistics center and the infrastructure structures to be inspected. The input data required is the location information of the target area and a detailed inspection plan. The terminal sends this information to the server. The output is flight plan data.

[0339] Step 2:

[0340] The server sends flight instructions to the drone based on the received flight plan data. The drone then begins flying toward the designated area. The input data is the flight plan data, and the output data is the drone's flight path information.

[0341] Step 3:

[0342] While flying over a designated area, the drone collects image data using a camera and sensor data using various sensors. The input data is the flight path, and the output data is the collected image data and sensor data.

[0343] Step 4:

[0344] The drones send the collected data to a server in real time. The server receives the data and stores it in a database. The input data is image data and sensor data from the drones, and the output data is stored data for analysis.

[0345] Step 5:

[0346] The server runs an image recognition algorithm on the received image data. The algorithm identifies abnormalities such as cracks, rust, and corrosion from the image data. The input data is the image data, and the output data is the analysis results of the abnormalities.

[0347] Step 6:

[0348] The server analyzes the sensor data and obtains information such as temperature, humidity, seismic intensity, and pressure. The input data is the sensor data, and the output data is the analysis result of the environmental information.

[0349] Step 7:

[0350] The server automatically generates inspection reports and inventory management reports based on the analysis results of image data and sensor data. The input data is the analysis results, and the output data is the report.

[0351] Step 8:

[0352] The server creates the generated report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. The input data is the report, and the output data is a notification email.

[0353] Step 9:

[0354] Users can check inspection results and inventory status in real time on a dashboard on a dedicated terminal. The input data is a report from the server, and the output data is the dashboard display content.

[0355] Step 10:

[0356] The emotion engine installed in the dedicated device analyzes the user's facial expressions and voice to recognize their emotional state. The input data is the user's facial expressions and voice data, and the output data is the emotion recognition results.

[0357] Step 11:

[0358] The server selects and displays an appropriate notification message depending on the user's emotional state. For example, if the user is feeling stressed, a message that provides reassurance is displayed. The input data is the emotion recognition result, and the output data is the display message.

[0359] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0360] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0361] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0362] [Second embodiment]

[0363] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0364] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0365] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0366] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0367] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0368] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0369] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0370] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0371] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0372] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0373] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0374] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0375] This invention is a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports, reducing the burden on engineers and enabling efficient and safe infrastructure inspections.

[0376] System Configuration

[0377] The system consists of the following main components:

[0378] 1. Aerial Vehicles

[0379] The drones collect images and sensor data of infrastructure structures as they fly.

[0380] 2. Server

[0381] The server receives, stores, and analyzes the data sent from the drone.

[0382] 3. User Interface Terminal

[0383] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0384] Program processing explanation

[0385] Data collection

[0386] The user uses a dedicated terminal to set the location information of the infrastructure to be inspected and a flight plan for the drone. Once the settings are complete, the terminal sends these instructions to the server. The server issues instructions to the drone based on the received information, causing the drone to fly automatically along the specified flight route. During flight, the drone uses cameras and various sensors to acquire image data and sensor data of the infrastructure, and sends this to the server in real time.

[0387] Data analysis

[0388] The server runs an image recognition algorithm on the received image data to identify abnormalities such as cracks, rust, and corrosion. The image recognition algorithm uses pre-trained pattern recognition technology to detect abnormalities with high accuracy. It also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity. This allows it to pinpoint abnormalities in detail.

[0389] Reporting and Notifications

[0390] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this inspection report in PDF format and emails it to managers and other relevant parties. It also has an alert function that ensures immediate notification if any important anomalies are discovered.

[0391] User Interface

[0392] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays the location of any abnormalities and detailed analysis results, and past inspection data can also be easily searched and referenced. This enables quick and efficient response.

[0393] Specific examples

[0394] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can then check the report on their device and develop a repair plan as needed.

[0395] This will enable efficient measures to address the deterioration of infrastructure and resolve problems caused by labor shortages.

[0396] The processing flow will be explained below.

[0397] Step 1:

[0398] The user uses a dedicated terminal to specify the infrastructure to be inspected and set up a flight plan, which includes the location of the inspection site, the drone's flight path, and the time of day.

[0399] Step 2:

[0400] The terminal sends the set flight plan to the server, which receives it and sends flight plan instructions to the drone.

[0401] Step 3:

[0402] The drone automatically flies a predetermined route according to the flight plan it is instructed to follow. During flight, the drone uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure.

[0403] Step 4:

[0404] The drones transmit the collected image data and sensor data in real time to a server, which receives the data and stores it in a database.

[0405] Step 5:

[0406] The server then runs image recognition algorithms on the received image data, which automatically detects abnormalities such as cracks, rust, and corrosion.

[0407] Step 6:

[0408] The server performs a detailed analysis of the detected anomalies, identifying their location, range, depth, etc., and generates the analysis results as text data.

[0409] Step 7:

[0410] The server also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, pressure, etc. If an abnormality is detected, it also identifies the location information, type and degree of the abnormality.

[0411] Step 8:

[0412] The server automatically generates an inspection report based on the analysis results, which includes detailed information on identified abnormalities and the priority of areas requiring repair.

[0413] Step 9:

[0414] The server outputs the generated inspection report in PDF format and notifies the administrator and other relevant parties by email. In addition, if any abnormalities are found to be urgent, an alert will be sent immediately.

[0415] Step 10:

[0416] Users can check inspection results in real time through a dashboard on a dedicated device, which displays details of abnormalities and analysis results, and can also easily refer to past inspection data.

[0417] Step 11:

[0418] Users can create repair plans via their terminals, which include the necessary work and schedules.

[0419] Step 12:

[0420] The server stores the details of the repair plan and updates the database so that it can be used at the next inspection.

[0421] This will enable the creation of a system that allows for efficient and safe infrastructure inspection and repair.

[0422] Example 1

[0423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Modern infrastructure structures are diverse, and their inspections require a huge amount of time and effort. Furthermore, traditional inspection methods require engineers to work at height or in dangerous environments, threatening their safety. Furthermore, manual inspections carry a high risk of overlooking or false positives, creating a need for efficient and highly accurate inspections.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0426] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for transmitting location information and a flight plan of the inspection target from a user terminal, means for executing an image recognition algorithm for identifying abnormalities in the infrastructure based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, and means for displaying the analysis results of the abnormalities to the user in real time. This enables efficient and high-precision inspection of infrastructure, reduces the burden on engineers, and improves safety.

[0427] An "aerial vehicle" is an unmanned flying device, such as a drone. It is primarily used to collect image data and sensor data over a wide area.

[0428] "Image data" refers to photographic images and videos taken by aerial vehicles, and is information used to visually grasp the appearance and condition of infrastructure structures.

[0429] "Sensor data" refers to physical data collected from various sensors (e.g., temperature sensors, humidity sensors, seismic intensity sensors) mounted on airborne vehicles, and is information used to numerically grasp environmental conditions and the status of infrastructure structures.

[0430] "User terminal" refers to an electronic device operated by a user, such as a tablet or laptop computer with dedicated software installed. It is used to set and transmit inspection target location information and flight plans.

[0431] A "server" is a computer that stores and processes data on a network, and receives, analyzes, stores, and reports data sent from airborne vehicles.

[0432] An "image recognition algorithm" is a technology for analyzing image data to detect specific patterns or abnormalities, and includes pattern recognition technology that uses AI and machine learning.

[0433] An "inspection report" is a document that summarizes the inspection results, including any identified abnormalities and the priority of necessary repair work. It is used to report to managers and other relevant parties.

[0434] "Real-time display" means that data is displayed to the user as soon as it is collected, allowing for rapid response in the field.

[0435] "Abnormal areas" refer to problems such as cracks, rust, and corrosion that exist in infrastructure structures, and are areas that require repair or restoration.

[0436] This invention is a system that uses an aerial vehicle to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. This system reduces the burden on engineers and enables efficient and safe infrastructure inspections.

[0437] System Configuration

[0438] The system consists of the following main components:

[0439] 1. Aerial Vehicles

[0440] Drones such as the DJI Phantom 4 and DJI Mavic 2 fly over the infrastructure and collect images and sensor data.

[0441] 2. Server

[0442] The server receives, stores, and analyzes the data, for example, using a cloud computing service such as AWS EC2 or Google Cloud.

[0443] 3. User Interface Terminal

[0444] A terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports. Tablets and laptops are commonly used as terminals.

[0445] Data collection

[0446] The user uses a dedicated device to input the location information of the infrastructure to be inspected and the flight plan into the drone. This is done using a tablet or laptop with a dedicated application installed. Once the settings are complete, the device sends these instructions to a server via Wi-Fi or 4G / 5G networks.

[0447] The server issues instructions to the drone based on the received information, causing it to fly automatically along the designated route. The drone is equipped with a camera and various sensors, and acquires image data and sensor data (temperature, humidity, seismic intensity, etc.) of infrastructure structures during flight, which is then sent to the server in real time.

[0448] Data analysis

[0449] When the server receives the transmitted image data, it analyzes the data using image recognition algorithms such as TensorFlow and OpenCV. The analysis algorithm uses pre-trained pattern recognition rules to identify abnormalities such as cracks, rust, and corrosion. At the same time, it also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, and seismic intensity. This allows for detailed identification of abnormalities.

[0450] Reporting and Notifications

[0451] Once the analysis is complete, the server automatically generates an inspection report using Python or JavaScript. The report contains detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this report in PDF format and emails it to administrators and other relevant parties using the SMTP protocol. If any serious anomalies are discovered, an alert can be sent immediately using the Twilio API or similar.

[0452] User Interface

[0453] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays abnormalities and detailed analysis results. This uses an interactive user interface built using React and Angular.js. Users can also easily search and reference past inspection data, enabling quick and efficient response.

[0454] Specific examples

[0455] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server analyzes the received data in real time and identifies cracks and corrosion. An inspection report is automatically generated based on the detection results and sent to the user's device, where the user can review the report and develop any necessary repair plans.

[0456] An example of a prompt sentence to be input to a specific generative AI model:

[0457] Prepare a bridge inspection report using the following information:

[0458] Name of the bridge: XX Bridge

[0459] Location information:35.710063, 139.8107

[0460] Acquired data: Image data (24 images in total), sensor data (temperature, humidity, seismic intensity)

[0461] Abnormalities: Cracks (5 places), rust (3 places)

[0462] Details of the abnormality: average crack length 0.5m, average rust area 10cm²

[0463] Based on this, generate an inspection report in PDF format.

[0464] The present invention makes it possible to efficiently address the deterioration of infrastructure, thereby resolving labor shortages and safety issues.

[0465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0466] Step 1: Setting up input data

[0467] The user uses a dedicated device to set the location information and flight plan for the infrastructure to be inspected. Input is made using a tablet or laptop, and detailed local coordinate information (latitude and longitude) and flight route are entered through a dedicated application. This generates instructions for the drone to navigate along the flight path.

[0468] Step 2: Submit your flight plan

[0469] The device transmits the location information of the inspection target and the flight plan set by the user to a server via Wi-Fi or 4G / 5G network. The server analyzes the received data and determines the route the drone should take. The generated data is then passed to the drone control API.

[0470] Step 3: Start flying the drone automatically

[0471] The server issues automatic flight instructions to the drone, using the drone's built-in flight control system. As the drone flies automatically along a set route, it uses its camera to capture image data of infrastructure structures and its sensors to collect physical data such as temperature, humidity, and seismic intensity. The collected data is sent to the server in real time for subsequent data analysis processing.

[0472] Step 4: Receiving and storing data

[0473] The server receives the image data and sensor data sent from the drone and first stores it in a database. The received data includes large amounts of image files and sensor output, which are later used for analytical processing. Amazon S3 or Google Cloud Storage are often used for server storage.

[0474] Step 5: Analyze imagery and sensor data

[0475] The server runs image recognition algorithms on the received and stored image data. For example, it uses TensorFlow or OpenCV to identify abnormalities such as cracks, rust, and corrosion. It analyzes the input data (image data) for image recognition processing and generates output results indicating abnormalities. These results are saved in JSON format and used to generate subsequent reports. At the same time, it also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity.

[0476] Step 6: Generate an inspection report

[0477] The server automatically generates a draft inspection report based on the results of the image and sensor data analysis. It generates the report in PDF format using Python's ReportLab library, among other tools. The report includes detailed information about identified anomalies and the priority of recommended repair work. The report is saved for future notification.

[0478] Step 7: Report Notifications and Alerts

[0479] The server notifies the administrator and relevant parties of the generated inspection report. The report is sent by email using the SMTP protocol. If a serious abnormality is detected, an alert is sent immediately via SMS or phone using the Twilio API. This makes it possible to respond quickly when a rapid response is required.

[0480] Step 8: Viewing in the User Interface

[0481] Users can view inspection results in real time via a dashboard on a dedicated device. The device dashboard provides an interactive interface built using React and Angular.js. Details of abnormalities and analysis results are visually displayed here, and past inspection data can be easily searched and referenced. This enables quick and efficient response.

[0482] (Application example 1)

[0483] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0484] Inspecting equipment within a factory requires inspecting high places and complex structural parts, and conventional methods place a heavy burden on personnel and pose safety issues. Furthermore, identifying abnormalities and creating inspection reports was done manually, which was inefficient and prone to errors. For this reason, there was a need for a system that could inspect factory equipment safely and efficiently and identify abnormalities in real time.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0486] In this invention, the server includes means for receiving image data and sensor data from equipment in the factory, means for executing an image recognition algorithm using AI technology to identify abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities and saving the generated report in a cloud server, and means for notifying an administrator of the generated inspection report in real time. This enables safe and efficient inspection of factory equipment, enables abnormalities to be identified in real time, and enables rapid response.

[0487] "Factory facilities" refers to the infrastructure used in factories, such as various machines, devices, piping, and electrical equipment.

[0488] "Image data" is data representing visual information captured using a camera or other imaging device.

[0489] "Sensor data" refers to data representing physical or environmental information obtained from various sensors.

[0490] "Means for receiving" refers to a combination of hardware and software for remotely obtaining data and processing the obtained data.

[0491] An "image recognition algorithm" is a computational procedure for analyzing image data and detecting specific patterns or anomalies within it.

[0492] "AI technology" is a technology that uses artificial intelligence to analyze data and perform pattern recognition.

[0493] "Automatic generation" refers to the process where a system automatically generates a report based on data without human intervention.

[0494] A "cloud server" is a remote virtual server used to store and process data over the Internet.

[0495] "Means of real-time notification" refers to a method or device that immediately transmits information to managers and relevant parties without delay when an abnormality is detected.

[0496] In this invention, we build a system for efficiently and safely inspecting equipment in factories. The system consists of a drone, a cloud server, and a smartphone app. To implement this invention, the following hardware and software are used.

[0497] Hardware

[0498] Drone: Equipped with a camera, temperature and humidity sensors, seismometer, and pressure sensor.

[0499] Cloud server: A remote server (typically AWS or Google Cloud) for storing and analyzing data.

[0500] User device: Smartphone or tablet (iOS or Android based).

[0501] software

[0502] Drone control software: Used for automated drone flight and data collection (e.g., DJI SDK).

[0503] Server communication library: Sends and receives data from the drone to the cloud server (e.g., HTTP communication library).

[0504] AI analysis library: Analyzes image data and sensor data to identify abnormalities (TensorFlow, OpenCV, etc.).

[0505] Report generation library: Automatically generate inspection reports (e.g., ReportLab).

[0506] Notification system: Provides real-time notifications to user devices (e.g., Firebase Cloud Messaging).

[0507] User interface: The application that runs on a smartphone or tablet.

[0508] System Operation

[0509] 1. Data Collection

[0510] Users use a smartphone app to enter information about the equipment they want to inspect and their inspection plan. Once the settings are complete, the cloud server transmits the information to the drone. The drone then flies automatically and uses its camera and sensors to collect image data and sensor data from the equipment. The data is then sent to the cloud server in real time.

[0511] 2. Data Analysis

[0512] The cloud server then runs an image recognition algorithm using AI technology on the received data. This algorithm detects abnormalities such as cracks, rust, and corrosion, and also identifies abnormalities in high places and complex structural parts. The results of this analysis are then stored on the cloud server.

[0513] 3. Report generation and notifications

[0514] Based on the analysis results, the cloud server automatically generates an inspection report and stores it in the cloud. If any significant abnormalities are detected, a notification system sends an alert to the user's device in real time. Users can check the inspection results on their smartphones or tablets and take any necessary measures.

[0515] Specific examples

[0516] For example, when conducting regular inspections of pipelines within a factory, the user launches the app and inputs the location information of the pipeline to be inspected and the inspection plan. The drone flies automatically according to instructions, capturing sensor data while photographing the area around the pipeline. The cloud server receives this data and performs AI analysis. If cracks, rust, etc. are identified, an inspection report is immediately generated and a notification is sent to the user's device. The user can check the report on the app and create a maintenance plan.

[0517] Prompt Sentence Examples

[0518] "We are thinking about a system that uses drones to detect abnormalities in factory equipment. In this system, drones fly automatically to collect image data and sensor data from the equipment, which is then analyzed by AI to identify any abnormalities. Specifically, please explain in detail what steps should be taken to build the system, including the hardware and software to be used."

[0519] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0520] Step 1:

[0521] The user launches the smartphone app and inputs information about the equipment they want to inspect (for example, the equipment's location and inspection plan). This information is used as input. The smartphone app then sends this input data to a cloud server. The data sent to the server becomes the output for creating an automatic flight plan for the drone.

[0522] Step 2:

[0523] The cloud server issues automatic flight instructions to the drone based on the received inspection plan. Once the flight plan is sent to the drone, the drone automatically flies along the designated route, collecting image data using its camera and sensor data using various sensors (temperature and humidity sensors, seismometers, pressure sensors). This becomes input data. The collected image data and sensor data are sent to the cloud server in real time and become output for analysis.

[0524] Step 3:

[0525] The cloud server runs an image recognition algorithm using AI technology on the image data and sensor data it receives in real time. Image data and sensor data are used as input data. The cloud server analyzes the image data using AI technology (for example, TensorFlow or OpenCV) to detect abnormalities such as cracks, rust, and corrosion. Detailed data on detected abnormalities is output.

[0526] Step 4:

[0527] The cloud server automatically generates an inspection report based on details of abnormalities detected by the image recognition algorithm. Detailed data on the abnormalities is used as input data. The server converts the generated report into PDF format and saves it in the cloud. The generated inspection report is the output.

[0528] Step 5:

[0529] The cloud server prepares the generated inspection report for notification to the administrator. The generated inspection report is used as input data. The server notifies the administrator via a notification system (e.g., Firebase Cloud Messaging) that the report is available to their smartphone or tablet. This is the output. If any significant anomalies are detected, an alert is sent immediately.

[0530] Step 6:

[0531] Users can check the inspection report through the application on their smartphone or tablet. The inspection report received after receiving the notification is used as input data. Users can check detailed abnormalities and inspection results through the app interface and plan necessary countermeasures. The checked report contents are output.

[0532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0533] This invention is a system that uses an aerial vehicle (drone) to collect and analyze image data and sensor data of infrastructure structures, generates inspection reports, and combines this with an emotion engine that recognizes the user's emotions. This system reduces the burden on engineers, enables efficient and safe infrastructure inspections, and presents information that takes user emotions into consideration.

[0534] System Configuration

[0535] The system consists of the following main components:

[0536] 1. Aerial Vehicles

[0537] The drones collect images and sensor data of infrastructure structures as they fly.

[0538] 2. Server

[0539] The server receives, stores, and analyzes the data sent from the drone.

[0540] 3. User Interface Terminal

[0541] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0542] 4. Emotion Engine

[0543] The emotion engine recognizes the user's emotions by analyzing their facial expressions and voice, and then displays appropriate information and notifications accordingly.

[0544] Program processing explanation

[0545] Data collection

[0546] The user uses a dedicated terminal to specify the infrastructure inspection target and set up a flight plan. The settings include the location information of the inspection location, the drone's flight path, and the time period. The terminal sends the set flight plan to the server. The server receives this and sends flight plan instructions to the drone. The drone follows the instructed flight plan and automatically flies the predetermined flight path. During flight, the drone uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure. The drone sends the collected image data and sensor data to the server in real time.

[0547] Data analysis

[0548] The server runs an image recognition algorithm on the received image data. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities, identifying the location, range, depth, etc. of the abnormality and generating the analysis results as text data. It also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, and pressure. If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0549] Report Generation

[0550] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and other relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0551] User Interface and Emotion Engine

[0552] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0553] Specific examples

[0554] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0555] This will enable efficient measures to deal with aging infrastructure and enable system operations that take user feelings into consideration.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] Using a dedicated terminal, the user selects the infrastructure to be inspected and sets up a flight plan, including details such as the target location, flight path, shooting interval, type of sensor to be used, and flight time.

[0559] Step 2:

[0560] The terminal sends the configured flight plan to the server, which receives the flight plan and transmits it to the drone.

[0561] Step 3:

[0562] The drone will automatically begin flying according to the received flight plan, and during the flight, it will use its cameras and sensors to collect image and sensor data.

[0563] Step 4:

[0564] The drone transmits the collected image data and sensor data to a server in real time.

[0565] Step 5:

[0566] The server runs an image recognition algorithm on the received image data, which uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[0567] Step 6:

[0568] The server identifies any abnormalities detected through image analysis and converts their location, range, and depth into text data as the analysis results.

[0569] Step 7:

[0570] The server also analyzes sensor data, such as seismic intensity, temperature, humidity, and pressure, and if an abnormality is detected, identifies the type and severity of the abnormality.

[0571] Step 8:

[0572] The server automatically generates an inspection report based on the analysis results, which includes text and diagrams and includes detailed information about any abnormalities and the priority of repairs.

[0573] Step 9:

[0574] The server notifies the administrator of the generated inspection report via email or SMS, with the report itself attached in PDF format. It also sends an immediate alert for any abnormalities that are deemed urgent.

[0575] Step 10:

[0576] Users can check the inspection and analysis results in real time through a dashboard on a dedicated device, which also displays detailed information on abnormalities and past inspection data.

[0577] Step 11:

[0578] The emotion engine recognizes the user's emotional state as they use the dashboard. The emotion engine analyzes the user's facial expressions and voice in real time to identify the user's emotional state.

[0579] Step 12:

[0580] The device adjusts the content and display of notification messages received by the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the device will display a notification message that gives a sense of security.

[0581] Step 13:

[0582] Users can review coordinated notifications and inspection reports and create necessary repair plans, which include specific work content and schedules.

[0583] Step 14:

[0584] The server stores the proposed repair plan in a database and updates the data so that it can be used at the next inspection.

[0585] This will enable efficient and safe infrastructure inspection and repair, while also providing information that takes user feelings into consideration.

[0586] Example 2

[0587] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] In conventional infrastructure inspection systems, the analysis and reporting of inspection results was done manually, which took time and effort and could lead to variations in the accuracy of the analysis. This also placed a heavy burden on engineers, posing safety and efficiency challenges. Furthermore, the system did not present information that took user emotions into consideration, potentially increasing user anxiety and stress.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0590] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, emotion analysis means for recognizing the user's emotional state and adjusting the notification message, and user interface means for displaying data in real time. This enables infrastructure inspection work to be performed efficiently and safely, reducing the burden on engineers. Furthermore, since information is presented taking the user's emotions into consideration, it is possible to reduce user anxiety and stress.

[0591] "Aerial vehicle" is a general term for unmanned aerial vehicles that fly over infrastructure structures such as stores and facilities to collect data.

[0592] "Image data" refers to visual information of infrastructure structures photographed by a camera mounted on an aerial vehicle.

[0593] "Sensor data" refers to physical property data such as temperature, humidity, pressure, and seismic intensity measured by various sensors mounted on airborne vehicles.

[0594] "Means for receiving" means the equipment or software functions for receiving and storing image data and sensor data transmitted from an airborne vehicle.

[0595] An "image recognition algorithm" is a program or method for automatically identifying abnormalities such as cracks, rust, and corrosion in received image data.

[0596] An "inspection report" is an automatically generated report based on abnormalities identified by an image recognition algorithm, and includes detailed information about the abnormalities and the priority of repair work.

[0597] The "means for notifying the administrator" refers to a communication means or software function for notifying the administrator or related parties of the generated inspection report in real time.

[0598] The "emotion analysis means" is a device or software function for analyzing the user's facial expressions and voice and recognizing the user's emotions.

[0599] "User interface means" refers to the function of a device or software that provides an interface that allows users to view and manipulate inspection data and results in real time.

[0600] This invention relates to a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. Furthermore, by combining this with an emotion engine that recognizes user emotions and presents appropriate information, it is possible to reduce the burden on engineers and perform infrastructure inspections efficiently and safely.

[0601] To implement this system, the following major hardware and software are used:

[0602] Hardware

[0603] 1. Aerial vehicle (drone): Equipped with a camera and multiple sensors (temperature and humidity sensors, pressure sensors, etc.), it has the ability to collect image data and sensor data.

[0604] 2. Server: A high-performance computing system for receiving, storing, and analyzing data sent from the drones.

[0605] 3. User interface terminal: A device operated by the user that provides an interface for issuing inspection instructions and checking analysis results and inspection reports.

[0606] software

[0607] 1. Image recognition algorithm: An image analysis program for detecting abnormalities such as cracks, rust, and corrosion. It is composed mainly of deep learning models.

[0608] 2. Data analysis software: A program that analyzes sensor data (temperature, humidity, pressure, seismic intensity, etc.) and evaluates physical properties.

[0609] 3. Emotion analysis engine: Software that analyzes the user's facial expressions and voice to recognize their emotional state.

[0610] 4. User interface software: An interface program designed to display data in real time and allow users to easily operate it.

[0611] Specific actions

[0612] The user uses a dedicated device to specify the infrastructure to be inspected and set up a flight plan, including the location of the inspection site, the drone's flight path, and the time of day. The flight path includes detailed instructions, such as covering both the top and bottom of a bridge.

[0613] The terminal sends the set flight plan to the server. The server receives it and sends flight plan instructions to the drone. The drone follows the instructed flight plan, automatically flying along a predetermined flight path while using cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure, and sends this data to the server in real time.

[0614] The server stores the received data in a database and runs an image recognition algorithm on the image data. This algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities to identify their location, range, depth, etc. It also analyzes sensor data to evaluate physical characteristics (temperature, humidity, seismic intensity, pressure, etc.). If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0615] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0616] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0617] Specific examples

[0618] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0619] Prompt Sentence Examples

[0620] 1. Drone flight plan configuration: "Please explain the process for configuring infrastructure location information and inspection plans (flight path, time slots, etc.) and sending them to the server."

[0621] 2. Data Analysis: Please explain in detail how the server analyzes the received image data and sensor data to detect any abnormalities.

[0622] 3. Report Generation: "Please explain in detail the steps by which the server automatically generates an inspection report based on the analysis results."

[0623] 4. How the Emotion Engine Works: "Please explain how the emotion engine works, recognizing the user's emotions and displaying appropriate information or notifications."

[0624] This will enable efficient measures to deal with aging infrastructure, and will also enable system operations that take user feelings into consideration.

[0625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] The user uses a dedicated terminal to specify the infrastructure structure (e.g., a bridge) to be inspected. Input includes the location information of the inspection location, the drone's flight path, and the flight time. The terminal compiles this information into a drone flight plan and generates data in JSON format. This data becomes the terminal's output.

[0628] Step 2:

[0629] The device sends the generated flight plan data to the server. The server receives this data and generates instruction data for the drone based on it. The instruction data includes the specific flight path, subject to be photographed, and timing of the photograph. The server's output is instruction data for the drone.

[0630] Step 3:

[0631] The server sends instruction data to the drone. Based on the received instruction data, the drone automatically begins flying according to the set flight path. During flight, the drone uses its on-board camera and sensors to collect image data and sensor data. This is the drone's output data.

[0632] Step 4:

[0633] The drone transmits the collected image data and sensor data to the server in real time. The server receives this data and stores it in a database. The stored image data and sensor data become the input for the server.

[0634] Step 5:

[0635] The server receives the stored image data as input and runs an image recognition algorithm. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. Information about the detected abnormalities (location, range, depth, etc.) is output from the server. The server also analyzes sensor data and evaluates physical properties (temperature, humidity, seismic intensity, pressure, etc.). The results of these analyses are also output from the server.

[0636] Step 6:

[0637] The server automatically generates an inspection report based on the results of the image recognition algorithm and sensor data analysis. The report contains detailed information about any identified abnormalities and the priority of repair work. The generated inspection report is the output of the server.

[0638] Step 7:

[0639] The server outputs the generated inspection report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. This is the output of the server.

[0640] Step 8:

[0641] The generated inspection report is displayed on the dashboard of the user's dedicated terminal. Through the dashboard, the user can check the report contents and analysis results in real time. The dashboard display is the output of the user's terminal.

[0642] Step 9:

[0643] The emotion analysis engine analyzes the face and voice of the user using the dashboard to recognize the user's emotional state. The analysis results are the output of the emotion analysis engine. For example, if the user is feeling anxious, the engine will select and display a notification message that will give a sense of security. This is the final output on the user's device.

[0644] (Application example 2)

[0645] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] In modern logistics centers, inventory management and inspection work remains highly manual and difficult to perform efficiently. Furthermore, there is a lack of support that takes into consideration the emotions of operators, which can lead to excessive stress in the work. Infrastructure inspection work also places a heavy burden on technicians, and it must be performed efficiently and safely. To solve these issues, automation using drones and a system that recognizes users' emotions and provides appropriate feedback are needed.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0648] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying a manager of the generated inspection report, an emotion engine that recognizes a user's emotions, means for displaying information and notifying users based on their emotions, means for a drone to fly over a logistics center area and collect inventory data, and means for processing the collected inventory data and generating an inventory management report. This enables efficient inventory management at logistics centers, efficient inspections of infrastructure structures, and support that takes user emotions into consideration.

[0649] An "aerial vehicle" is an unmanned mechanical device capable of flying in the air, and in this context refers specifically to a drone.

[0650] "Image data" is data that includes visual information acquired by a photographic device such as a drone camera.

[0651] "Sensor data" refers to data about the physical characteristics of the environment and materials collected by various sensors mounted on drones.

[0652] An "image recognition algorithm" is a calculation method for extracting the characteristics of an object from image data and identifying abnormal areas.

[0653] An "inspection report" refers to a report generated based on the results of image recognition algorithms and sensor data analysis.

[0654] The "notification means" refers to a method or device for transmitting the generated inspection report or inventory control report to relevant parties or managers.

[0655] An "emotion engine" is a technology that analyzes a user's facial expressions and voice data to recognize their emotional state.

[0656] "Means for drones to fly over logistics center areas and collect inventory data" refers to a method for drones to fly autonomously over designated areas and obtain information on inventory items.

[0657] An "inventory management report" is a report that lists inventory status and abnormalities generated based on data collected by drones.

[0658] This invention is a system that uses aerial vehicles (drones) to collect and analyze data from infrastructure structures and logistics centers, improving the efficiency of inspections and inventory management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system displays and notifies information that takes the user's emotions into consideration. A specific embodiment of this system is described below.

[0659] Data collection

[0660] The server sends instructions to the drone based on a flight plan sent from the user interface terminal. The drone flies over designated infrastructure structures and logistics center areas, collecting image data using its camera and sensor data using various sensors. The collected data is sent to the server in real time.

[0661] Data analysis

[0662] The server uses image data sent from the drone to run image recognition algorithms and analyze abnormalities in infrastructure and inventory information within the logistics center. To detect abnormalities, it uses pattern recognition rules to identify cracks, rust, corrosion, and other abnormalities. It also analyzes sensor data and evaluates information such as temperature, humidity, seismic intensity, and pressure. Based on the results of this analysis, it automatically generates inspection reports for infrastructure and inventory management reports for the logistics center.

[0663] Reporting and Notifications

[0664] The server automatically generates a report of the analysis results, outputs the report in PDF format, and notifies administrators and other relevant parties by email. If a significant abnormality or inventory abnormality is discovered, an alert is sent immediately.

[0665] User Interface and Emotion Engine

[0666] Using a dedicated user interface terminal, users can specify inspection targets and inventory management areas and check the results in real time. The terminal displays detailed inspection results and inventory status, and past data can also be easily referenced. In addition, an emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling stressed, the server will select a reassuring notification message and display it to the user.

[0667] Specific examples

[0668] For example, when conducting an inventory inspection at a logistics center, the user sets the location information of inventory shelves and an inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the inventory shelves and collecting sensor data. The server receives this data in real time and uses image recognition algorithms to check inventory and detect abnormalities. Based on the detection results, the server automatically generates an inventory management report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately adjusted by the emotion engine.

[0669] Prompt Sentence Examples

[0670] "We will begin an inspection of inventory shelf area A. Please fly the drone into the area and scan the product barcodes."

[0671] "There is an abnormality in the data being analyzed. The upper shelf of inventory shelf A is misaligned and needs to be checked."

[0672] "Stress response detected. Providing advice for efficient stock replenishment."

[0673] This will enable efficient inventory management at logistics centers and inspection of infrastructure structures, as well as support that takes user emotions into consideration.

[0674] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0675] Step 1:

[0676] Users log in using a dedicated terminal and specify the inventory area of ​​the logistics center and the infrastructure structures to be inspected. The input data required is the location information of the target area and a detailed inspection plan. The terminal sends this information to the server. The output is flight plan data.

[0677] Step 2:

[0678] The server sends flight instructions to the drone based on the received flight plan data. The drone then begins flying toward the designated area. The input data is the flight plan data, and the output data is the drone's flight path information.

[0679] Step 3:

[0680] While flying over a designated area, the drone collects image data using a camera and sensor data using various sensors. The input data is the flight path, and the output data is the collected image data and sensor data.

[0681] Step 4:

[0682] The drones send the collected data to a server in real time. The server receives the data and stores it in a database. The input data is image data and sensor data from the drones, and the output data is stored data for analysis.

[0683] Step 5:

[0684] The server runs an image recognition algorithm on the received image data. The algorithm identifies abnormalities such as cracks, rust, and corrosion from the image data. The input data is the image data, and the output data is the analysis results of the abnormalities.

[0685] Step 6:

[0686] The server analyzes the sensor data and obtains information such as temperature, humidity, seismic intensity, and pressure. The input data is the sensor data, and the output data is the analysis result of the environmental information.

[0687] Step 7:

[0688] The server automatically generates inspection reports and inventory management reports based on the analysis results of image data and sensor data. The input data is the analysis results, and the output data is the report.

[0689] Step 8:

[0690] The server creates the generated report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. The input data is the report, and the output data is a notification email.

[0691] Step 9:

[0692] Users can check inspection results and inventory status in real time on a dashboard on a dedicated terminal. The input data is a report from the server, and the output data is the dashboard display content.

[0693] Step 10:

[0694] The emotion engine installed in the dedicated device analyzes the user's facial expressions and voice to recognize their emotional state. The input data is the user's facial expressions and voice data, and the output data is the emotion recognition results.

[0695] Step 11:

[0696] The server selects and displays an appropriate notification message depending on the user's emotional state. For example, if the user is feeling stressed, a message that provides reassurance is displayed. The input data is the emotion recognition result, and the output data is the display message.

[0697] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0698] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0699] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0700] [Third embodiment]

[0701] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0702] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0703] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0704] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0705] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0707] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0708] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0709] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0710] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0711] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0712] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0713] This invention is a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports, reducing the burden on engineers and enabling efficient and safe infrastructure inspections.

[0714] System Configuration

[0715] The system consists of the following main components:

[0716] 1. Aerial Vehicles

[0717] The drones collect images and sensor data of infrastructure structures as they fly.

[0718] 2. Server

[0719] The server receives, stores, and analyzes the data sent from the drone.

[0720] 3. User Interface Terminal

[0721] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0722] Program processing explanation

[0723] Data collection

[0724] The user uses a dedicated terminal to set the location information of the infrastructure to be inspected and a flight plan for the drone. Once the settings are complete, the terminal sends these instructions to the server. The server issues instructions to the drone based on the received information, causing the drone to fly automatically along the specified flight route. During flight, the drone uses cameras and various sensors to acquire image data and sensor data of the infrastructure, and sends this to the server in real time.

[0725] Data analysis

[0726] The server runs an image recognition algorithm on the received image data to identify abnormalities such as cracks, rust, and corrosion. The image recognition algorithm uses pre-trained pattern recognition technology to detect abnormalities with high accuracy. It also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity. This allows it to pinpoint abnormalities in detail.

[0727] Reporting and Notifications

[0728] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this inspection report in PDF format and emails it to managers and other relevant parties. It also has an alert function that ensures immediate notification if any important anomalies are discovered.

[0729] User Interface

[0730] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays the location of any abnormalities and detailed analysis results, and past inspection data can also be easily searched and referenced. This enables quick and efficient response.

[0731] Specific examples

[0732] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can then check the report on their device and develop a repair plan as needed.

[0733] This will enable efficient measures to address the deterioration of infrastructure and resolve problems caused by labor shortages.

[0734] The processing flow will be explained below.

[0735] Step 1:

[0736] The user uses a dedicated terminal to specify the infrastructure to be inspected and set up a flight plan, which includes the location of the inspection site, the drone's flight path, and the time of day.

[0737] Step 2:

[0738] The terminal sends the set flight plan to the server, which receives it and sends flight plan instructions to the drone.

[0739] Step 3:

[0740] The drone automatically flies a predetermined route according to a flight plan, and during flight, it uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure.

[0741] Step 4:

[0742] The drones transmit the collected image data and sensor data in real time to a server, which receives the data and stores it in a database.

[0743] Step 5:

[0744] The server then runs image recognition algorithms on the received image data, which automatically detects abnormalities such as cracks, rust, and corrosion.

[0745] Step 6:

[0746] The server performs a detailed analysis of the detected anomalies, identifying their location, range, depth, etc., and generates the analysis results as text data.

[0747] Step 7:

[0748] The server also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, pressure, etc. If an abnormality is detected, it also identifies the location, type, and severity of the abnormality.

[0749] Step 8:

[0750] The server automatically generates an inspection report based on the analysis results, which includes detailed information on identified abnormalities and the priority of areas requiring repair.

[0751] Step 9:

[0752] The server outputs the generated inspection report in PDF format and notifies the administrator and other relevant parties by email. In addition, if any abnormalities are found to be urgent, an alert will be sent immediately.

[0753] Step 10:

[0754] Users can check inspection results in real time through a dashboard on a dedicated device, which displays details of abnormalities and analysis results, and can also easily refer to past inspection data.

[0755] Step 11:

[0756] Users can create repair plans via their terminals, which include the necessary work and schedules.

[0757] Step 12:

[0758] The server stores the details of the repair plan and updates the database so that it can be used at the next inspection.

[0759] This will enable the creation of a system that allows for efficient and safe infrastructure inspection and repair.

[0760] Example 1

[0761] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0762] Modern infrastructure structures are diverse, and their inspections require a huge amount of time and effort. Furthermore, traditional inspection methods require engineers to work at height or in dangerous environments, threatening their safety. Furthermore, manual inspections carry a high risk of overlooking or false positives, creating a need for efficient and highly accurate inspections.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0764] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for transmitting location information and a flight plan of the inspection target from a user terminal, means for executing an image recognition algorithm for identifying abnormalities in the infrastructure based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, and means for displaying the analysis results of the abnormalities to the user in real time. This enables efficient and high-precision inspection of infrastructure, reduces the burden on engineers, and improves safety.

[0765] An "aerial vehicle" is an unmanned flying device, such as a drone. It is primarily used to collect image data and sensor data over a wide area.

[0766] "Image data" refers to photographic images and videos taken by aerial vehicles, and is information used to visually grasp the appearance and condition of infrastructure structures.

[0767] "Sensor data" refers to physical data collected from various sensors (e.g., temperature sensors, humidity sensors, seismic intensity sensors) mounted on airborne vehicles, and is information used to numerically grasp environmental conditions and the status of infrastructure structures.

[0768] "User terminal" refers to an electronic device operated by a user, such as a tablet or laptop computer with dedicated software installed. It is used to set and transmit inspection target location information and flight plans.

[0769] A "server" is a computer that stores and processes data on a network, and receives, analyzes, stores, and reports data sent from airborne vehicles.

[0770] An "image recognition algorithm" is a technology for analyzing image data to detect specific patterns or abnormalities, and includes pattern recognition technology that uses AI and machine learning.

[0771] An "inspection report" is a document that summarizes the inspection results, including any identified abnormalities and the priority of necessary repair work. It is used to report to managers and other relevant parties.

[0772] "Real-time display" means that data is displayed to the user as soon as it is collected, allowing for rapid response in the field.

[0773] "Abnormal areas" refer to problems such as cracks, rust, and corrosion that exist in infrastructure structures, and are areas that require repair or restoration.

[0774] This invention is a system that uses an aerial vehicle to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. This system reduces the burden on engineers and enables efficient and safe infrastructure inspections.

[0775] System Configuration

[0776] The system consists of the following main components:

[0777] 1. Aerial Vehicles

[0778] Drones such as the DJI Phantom 4 and DJI Mavic 2 fly over the infrastructure and collect images and sensor data.

[0779] 2. Server

[0780] The server receives, stores, and analyzes the data, for example, using a cloud computing service such as AWS EC2 or Google Cloud.

[0781] 3. User Interface Terminal

[0782] A terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports. Tablets and laptops are commonly used as terminals.

[0783] Data collection

[0784] The user uses a dedicated device to input the location information of the infrastructure to be inspected and the flight plan into the drone. This is done using a tablet or laptop with a dedicated application installed. Once the settings are complete, the device sends these instructions to a server via Wi-Fi or 4G / 5G networks.

[0785] The server issues instructions to the drone based on the received information, causing it to fly automatically along the designated route. The drone is equipped with a camera and various sensors, and acquires image data and sensor data (temperature, humidity, seismic intensity, etc.) of infrastructure structures during flight, which is then sent to the server in real time.

[0786] Data analysis

[0787] When the server receives the transmitted image data, it analyzes the data using image recognition algorithms such as TensorFlow and OpenCV. The analysis algorithm uses pre-trained pattern recognition rules to identify abnormalities such as cracks, rust, and corrosion. At the same time, it also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, and seismic intensity. This allows for detailed identification of abnormalities.

[0788] Reporting and Notifications

[0789] Once the analysis is complete, the server automatically generates an inspection report using Python or JavaScript. The report contains detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this report in PDF format and emails it to administrators and other relevant parties using the SMTP protocol. If any serious anomalies are discovered, an alert can be sent immediately using the Twilio API or similar.

[0790] User Interface

[0791] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays abnormalities and detailed analysis results. This uses an interactive user interface built using React and Angular.js. Users can also easily search and reference past inspection data, enabling quick and efficient response.

[0792] Specific examples

[0793] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server analyzes the received data in real time and identifies cracks and corrosion. An inspection report is automatically generated based on the detection results and sent to the user's device, where the user can review the report and develop any necessary repair plans.

[0794] An example of a prompt sentence to be input to a specific generative AI model:

[0795] Prepare a bridge inspection report using the following information:

[0796] Name of the bridge: XX Bridge

[0797] Location information:35.710063, 139.8107

[0798] Acquired data: Image data (24 images in total), sensor data (temperature, humidity, seismic intensity)

[0799] Abnormalities: Cracks (5 places), rust (3 places)

[0800] Details of the abnormality: average crack length 0.5m, average rust area 10cm²

[0801] Based on this, generate an inspection report in PDF format.

[0802] The present invention makes it possible to efficiently address the deterioration of infrastructure, thereby resolving labor shortages and safety issues.

[0803] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0804] Step 1: Setting up input data

[0805] The user uses a dedicated device to set the location information and flight plan for the infrastructure to be inspected. Input is made using a tablet or laptop, and detailed local coordinate information (latitude and longitude) and flight route are entered through a dedicated application. This generates instructions for the drone to navigate along the flight path.

[0806] Step 2: Submit your flight plan

[0807] The device transmits the location information of the inspection target and the flight plan set by the user to a server via Wi-Fi or 4G / 5G network. The server analyzes the received data and determines the route the drone should take. The generated data is then passed to the drone control API.

[0808] Step 3: Start flying the drone automatically

[0809] The server issues automatic flight instructions to the drone, using the drone's built-in flight control system. As the drone flies automatically along a set route, it uses its camera to capture image data of infrastructure structures and its sensors to collect physical data such as temperature, humidity, and seismic intensity. The collected data is sent to the server in real time for subsequent data analysis processing.

[0810] Step 4: Receiving and storing data

[0811] The server receives the image data and sensor data sent from the drone and first stores it in a database. The received data includes large amounts of image files and sensor output, which are later used for analytical processing. Amazon S3 or Google Cloud Storage are often used for server storage.

[0812] Step 5: Analyze imagery and sensor data

[0813] The server runs image recognition algorithms on the received and stored image data. For example, it uses TensorFlow or OpenCV to identify abnormalities such as cracks, rust, and corrosion. It analyzes the input data (image data) for image recognition processing and generates output results indicating abnormalities. These results are saved in JSON format and used to generate subsequent reports. At the same time, it also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity.

[0814] Step 6: Generate an inspection report

[0815] The server automatically generates a draft inspection report based on the results of the image and sensor data analysis. It generates the report in PDF format using Python's ReportLab library, among other tools. The report includes detailed information about identified anomalies and the priority of recommended repair work. The report is saved for future notification.

[0816] Step 7: Report Notifications and Alerts

[0817] The server notifies the administrator and relevant parties of the generated inspection report. The report is sent by email using the SMTP protocol. If a serious abnormality is detected, an alert is sent immediately via SMS or phone using the Twilio API. This makes it possible to respond quickly when a rapid response is required.

[0818] Step 8: Viewing in the User Interface

[0819] Users can view inspection results in real time via a dashboard on a dedicated device. The device dashboard provides an interactive interface built using React and Angular.js. Details of abnormalities and analysis results are visually displayed here, and past inspection data can be easily searched and referenced. This enables quick and efficient response.

[0820] (Application example 1)

[0821] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0822] Inspecting equipment within a factory requires inspecting high places and complex structural parts, and conventional methods place a heavy burden on personnel and pose safety issues. Furthermore, identifying abnormalities and creating inspection reports was done manually, which was inefficient and prone to errors. For this reason, there was a need for a system that could inspect factory equipment safely and efficiently and identify abnormalities in real time.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0824] In this invention, the server includes means for receiving image data and sensor data from equipment in the factory, means for executing an image recognition algorithm using AI technology to identify abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities and saving the generated report in a cloud server, and means for notifying an administrator of the generated inspection report in real time. This enables safe and efficient inspection of factory equipment, enables abnormalities to be identified in real time, and enables rapid response.

[0825] "Factory facilities" refers to the infrastructure used in factories, such as various machines, devices, piping, and electrical equipment.

[0826] "Image data" is data representing visual information captured using a camera or other imaging device.

[0827] "Sensor data" refers to data representing physical or environmental information obtained from various sensors.

[0828] "Means for receiving" refers to a combination of hardware and software for remotely obtaining data and processing the obtained data.

[0829] An "image recognition algorithm" is a computational procedure for analyzing image data and detecting specific patterns or anomalies within it.

[0830] "AI technology" is a technology that uses artificial intelligence to analyze data and perform pattern recognition.

[0831] "Automatic generation" refers to the process where a system automatically generates a report based on data without human intervention.

[0832] A "cloud server" is a remote virtual server used to store and process data over the Internet.

[0833] "Means of real-time notification" refers to a method or device that immediately transmits information to managers and relevant parties without delay when an abnormality is detected.

[0834] In this invention, we build a system for efficiently and safely inspecting equipment in factories. The system consists of a drone, a cloud server, and a smartphone app. To implement this invention, the following hardware and software are used.

[0835] Hardware

[0836] Drone: Equipped with a camera, temperature and humidity sensors, seismometer, and pressure sensor.

[0837] Cloud server: A remote server (typically AWS or Google Cloud) for storing and analyzing data.

[0838] User device: Smartphone or tablet (iOS or Android based).

[0839] software

[0840] Drone control software: Used for automated drone flight and data collection (e.g., DJI SDK).

[0841] Server communication library: Sends and receives data from the drone to the cloud server (e.g., HTTP communication library).

[0842] AI analysis library: Analyzes image data and sensor data to identify abnormalities (TensorFlow, OpenCV, etc.).

[0843] Report generation library: Automatically generate inspection reports (e.g., ReportLab).

[0844] Notification system: Provides real-time notifications to user devices (e.g., Firebase Cloud Messaging).

[0845] User interface: The application that runs on a smartphone or tablet.

[0846] System Operation

[0847] 1. Data Collection

[0848] Users use a smartphone app to enter information about the equipment they want to inspect and their inspection plan. Once the settings are complete, the cloud server transmits the information to the drone. The drone then flies automatically and uses its camera and sensors to collect image data and sensor data from the equipment. The data is then sent to the cloud server in real time.

[0849] 2. Data Analysis

[0850] The cloud server then runs an image recognition algorithm using AI technology on the received data. This algorithm detects abnormalities such as cracks, rust, and corrosion, and also identifies abnormalities in high places and complex structural parts. The results of this analysis are then stored on the cloud server.

[0851] 3. Report generation and notifications

[0852] Based on the analysis results, the cloud server automatically generates an inspection report and stores it in the cloud. If any significant abnormalities are detected, a notification system sends an alert to the user's device in real time. Users can check the inspection results on their smartphones or tablets and take any necessary measures.

[0853] Specific examples

[0854] For example, when conducting regular inspections of pipelines within a factory, the user launches the app and inputs the location information of the pipeline to be inspected and the inspection plan. The drone flies automatically according to instructions, capturing sensor data while photographing the area around the pipeline. The cloud server receives this data and performs AI analysis. If cracks, rust, etc. are identified, an inspection report is immediately generated and a notification is sent to the user's device. The user can check the report on the app and create a maintenance plan.

[0855] Prompt Sentence Examples

[0856] "We are thinking about a system that uses drones to detect abnormalities in factory equipment. In this system, drones fly automatically to collect image data and sensor data from the equipment, which is then analyzed by AI to identify any abnormalities. Specifically, please explain in detail what steps should be taken to build the system, including the hardware and software to be used."

[0857] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0858] Step 1:

[0859] The user launches the smartphone app and inputs information about the equipment they want to inspect (for example, the equipment's location and inspection plan). This information is used as input. The smartphone app then sends this input data to a cloud server. The data sent to the server becomes the output for creating an automatic flight plan for the drone.

[0860] Step 2:

[0861] The cloud server issues automatic flight instructions to the drone based on the received inspection plan. Once the flight plan is sent to the drone, the drone automatically flies along the designated route, collecting image data using its camera and sensor data using various sensors (temperature and humidity sensors, seismometers, pressure sensors). This becomes input data. The collected image data and sensor data are sent to the cloud server in real time and become output for analysis.

[0862] Step 3:

[0863] The cloud server runs an image recognition algorithm using AI technology on the image data and sensor data it receives in real time. Image data and sensor data are used as input data. The cloud server analyzes the image data using AI technology (for example, TensorFlow or OpenCV) to detect abnormalities such as cracks, rust, and corrosion. Detailed data on detected abnormalities is output.

[0864] Step 4:

[0865] The cloud server automatically generates an inspection report based on details of abnormalities detected by the image recognition algorithm. Detailed data on the abnormalities is used as input data. The server converts the generated report into PDF format and saves it in the cloud. The generated inspection report is the output.

[0866] Step 5:

[0867] The cloud server prepares the generated inspection report for notification to the administrator. The generated inspection report is used as input data. The server notifies the administrator via a notification system (e.g., Firebase Cloud Messaging) that the report is available to their smartphone or tablet. This is the output. If any significant anomalies are detected, an alert is sent immediately.

[0868] Step 6:

[0869] Users can check the inspection report through the application on their smartphone or tablet. The inspection report received after receiving the notification is used as input data. Users can check detailed abnormalities and inspection results through the app interface and plan necessary countermeasures. The checked report contents are output.

[0870] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0871] This invention is a system that uses an aerial vehicle (drone) to collect and analyze image data and sensor data of infrastructure structures, generates inspection reports, and combines this with an emotion engine that recognizes the user's emotions. This system reduces the burden on engineers, enables efficient and safe infrastructure inspections, and presents information that takes user emotions into consideration.

[0872] System Configuration

[0873] The system consists of the following main components:

[0874] 1. Aerial Vehicles

[0875] The drones collect images and sensor data of infrastructure structures as they fly.

[0876] 2. Server

[0877] The server receives, stores, and analyzes the data sent from the drone.

[0878] 3. User Interface Terminal

[0879] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[0880] 4. Emotion Engine

[0881] The emotion engine recognizes the user's emotions by analyzing their facial expressions and voice, and then displays appropriate information and notifications accordingly.

[0882] Program processing explanation

[0883] Data collection

[0884] The user uses a dedicated terminal to specify the infrastructure inspection target and set up a flight plan. The settings include the location information of the inspection location, the drone's flight path, and the time period. The terminal sends the set flight plan to the server. The server receives this and sends flight plan instructions to the drone. The drone follows the instructed flight plan and automatically flies the predetermined flight path. During flight, the drone uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure. The drone sends the collected image data and sensor data to the server in real time.

[0885] Data analysis

[0886] The server runs an image recognition algorithm on the received image data. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities, identifying the location, range, depth, etc. of the abnormality and generating the analysis results as text data. It also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, and pressure. If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0887] Report Generation

[0888] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and other relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0889] User Interface and Emotion Engine

[0890] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0891] Specific examples

[0892] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0893] This will enable efficient measures to deal with aging infrastructure and enable system operations that take user feelings into consideration.

[0894] The processing flow will be explained below.

[0895] Step 1:

[0896] Using a dedicated terminal, the user selects the infrastructure to be inspected and sets up a flight plan, including details such as the target location, flight path, shooting interval, type of sensor to be used, and flight time.

[0897] Step 2:

[0898] The terminal sends the configured flight plan to the server, which receives the flight plan and transmits it to the drone.

[0899] Step 3:

[0900] The drone will automatically begin flying according to the received flight plan, and during the flight, it will use its cameras and sensors to collect image and sensor data.

[0901] Step 4:

[0902] The drone transmits the collected image data and sensor data to a server in real time.

[0903] Step 5:

[0904] The server runs an image recognition algorithm on the received image data, which uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[0905] Step 6:

[0906] The server identifies any abnormalities detected through image analysis and converts their location, range, and depth into text data as the analysis results.

[0907] Step 7:

[0908] The server also analyzes sensor data, such as seismic intensity, temperature, humidity, and pressure, and if an abnormality is detected, identifies the type and severity of the abnormality.

[0909] Step 8:

[0910] The server automatically generates an inspection report based on the analysis results, which includes text and diagrams and includes detailed information about any abnormalities and the priority of repairs.

[0911] Step 9:

[0912] The server notifies the administrator of the generated inspection report via email or SMS, with the report itself attached in PDF format. It also sends an immediate alert for any abnormalities that are deemed urgent.

[0913] Step 10:

[0914] Users can check the inspection and analysis results in real time through a dashboard on a dedicated device, which also displays detailed information on abnormalities and past inspection data.

[0915] Step 11:

[0916] The emotion engine recognizes the user's emotional state as they use the dashboard. The emotion engine analyzes the user's facial expressions and voice in real time to identify the user's emotional state.

[0917] Step 12:

[0918] The device adjusts the content and display of notification messages received by the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the device will display a notification message that gives a sense of security.

[0919] Step 13:

[0920] Users can review coordinated notifications and inspection reports and create necessary repair plans, which include specific work content and schedules.

[0921] Step 14:

[0922] The server stores the proposed repair plan in a database and updates the data so that it can be used at the next inspection.

[0923] This will enable efficient and safe infrastructure inspection and repair, while also providing information that takes user feelings into consideration.

[0924] Example 2

[0925] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] In conventional infrastructure inspection systems, the analysis and reporting of inspection results was done manually, which took time and effort and could lead to variations in the accuracy of the analysis. This also placed a heavy burden on engineers, posing safety and efficiency challenges. Furthermore, the system did not present information that took user emotions into consideration, potentially increasing user anxiety and stress.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0928] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, emotion analysis means for recognizing the user's emotional state and adjusting the notification message, and user interface means for displaying data in real time. This enables infrastructure inspection work to be performed efficiently and safely, reducing the burden on engineers. Furthermore, since information is presented taking the user's emotions into consideration, it is possible to reduce user anxiety and stress.

[0929] "Aerial vehicle" is a general term for unmanned aerial vehicles that fly over infrastructure structures such as stores and facilities to collect data.

[0930] "Image data" refers to visual information of infrastructure structures photographed by a camera mounted on an aerial vehicle.

[0931] "Sensor data" refers to physical property data such as temperature, humidity, pressure, and seismic intensity measured by various sensors mounted on airborne vehicles.

[0932] "Means for receiving" means the equipment or software functions for receiving and storing image data and sensor data transmitted from an airborne vehicle.

[0933] An "image recognition algorithm" is a program or method for automatically identifying abnormalities such as cracks, rust, and corrosion in received image data.

[0934] An "inspection report" is an automatically generated report based on abnormalities identified by an image recognition algorithm, and includes detailed information about the abnormalities and the priority of repair work.

[0935] The "means for notifying the administrator" refers to a communication means or software function for notifying the administrator or related parties of the generated inspection report in real time.

[0936] The "emotion analysis means" is a device or software function for analyzing the user's facial expressions and voice and recognizing the user's emotions.

[0937] "User interface means" refers to the function of a device or software that provides an interface that allows users to view and manipulate inspection data and results in real time.

[0938] This invention relates to a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. Furthermore, by combining this with an emotion engine that recognizes user emotions and presents appropriate information, it is possible to reduce the burden on engineers and perform infrastructure inspections efficiently and safely.

[0939] To implement this system, the following major hardware and software are used:

[0940] Hardware

[0941] 1. Aerial vehicle (drone): Equipped with a camera and multiple sensors (temperature and humidity sensors, pressure sensors, etc.), it has the ability to collect image data and sensor data.

[0942] 2. Server: A high-performance computing system for receiving, storing, and analyzing data sent from the drones.

[0943] 3. User interface terminal: A device operated by the user that provides an interface for issuing inspection instructions and checking analysis results and inspection reports.

[0944] software

[0945] 1. Image recognition algorithm: An image analysis program for detecting abnormalities such as cracks, rust, and corrosion. It is composed mainly of deep learning models.

[0946] 2. Data analysis software: A program that analyzes sensor data (temperature, humidity, pressure, seismic intensity, etc.) and evaluates physical properties.

[0947] 3. Emotion analysis engine: Software that analyzes the user's facial expressions and voice to recognize their emotional state.

[0948] 4. User interface software: An interface program designed to display data in real time and allow users to easily operate it.

[0949] Specific actions

[0950] The user uses a dedicated device to specify the infrastructure to be inspected and set up a flight plan, including the location of the inspection site, the drone's flight path, and the time of day. The flight path includes detailed instructions, such as covering both the top and bottom of a bridge.

[0951] The terminal sends the set flight plan to the server. The server receives it and sends flight plan instructions to the drone. The drone follows the instructed flight plan, automatically flying along a predetermined flight path while using cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure, and sends this data to the server in real time.

[0952] The server stores the received data in a database and runs an image recognition algorithm on the image data. This algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities to identify their location, range, depth, etc. It also analyzes sensor data to evaluate physical characteristics (temperature, humidity, seismic intensity, pressure, etc.). If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[0953] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[0954] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[0955] Specific examples

[0956] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[0957] Prompt Sentence Examples

[0958] 1. Drone flight plan configuration: "Please explain the process for configuring infrastructure location information and inspection plans (flight path, time slots, etc.) and sending them to the server."

[0959] 2. Data Analysis: Please explain in detail how the server analyzes the received image data and sensor data to detect any abnormalities.

[0960] 3. Report Generation: "Please explain in detail the steps by which the server automatically generates an inspection report based on the analysis results."

[0961] 4. How the Emotion Engine Works: "Please explain how the emotion engine works, recognizing the user's emotions and displaying appropriate information or notifications."

[0962] This will enable efficient measures to deal with aging infrastructure, and will also enable system operations that take user feelings into consideration.

[0963] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0964] Step 1:

[0965] The user uses a dedicated terminal to specify the infrastructure structure (e.g., a bridge) to be inspected. Input includes the location information of the inspection location, the drone's flight path, and the flight time. The terminal compiles this information into a drone flight plan and generates data in JSON format. This data becomes the terminal's output.

[0966] Step 2:

[0967] The device sends the generated flight plan data to the server. The server receives this data and generates instruction data for the drone based on it. The instruction data includes the specific flight path, subject to be photographed, and timing of the photograph. The server's output is instruction data for the drone.

[0968] Step 3:

[0969] The server sends instruction data to the drone. Based on the received instruction data, the drone automatically begins flying according to the set flight path. During flight, the drone uses its on-board camera and sensors to collect image data and sensor data. This is the drone's output data.

[0970] Step 4:

[0971] The drone transmits the collected image data and sensor data to the server in real time. The server receives this data and stores it in a database. The stored image data and sensor data become the input for the server.

[0972] Step 5:

[0973] The server receives the stored image data as input and runs an image recognition algorithm. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. Information about the detected abnormalities (location, range, depth, etc.) is output from the server. The server also analyzes sensor data and evaluates physical properties (temperature, humidity, seismic intensity, pressure, etc.). The results of these analyses are also output from the server.

[0974] Step 6:

[0975] The server automatically generates an inspection report based on the results of the image recognition algorithm and sensor data analysis. The report contains detailed information about any identified abnormalities and the priority of repair work. The generated inspection report is the output of the server.

[0976] Step 7:

[0977] The server outputs the generated inspection report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. This is the output of the server.

[0978] Step 8:

[0979] The generated inspection report is displayed on the dashboard of the user's dedicated terminal. Through the dashboard, the user can check the report contents and analysis results in real time. The dashboard display is the output of the user's terminal.

[0980] Step 9:

[0981] The emotion analysis engine analyzes the face and voice of the user using the dashboard to recognize the user's emotional state. The analysis results are the output of the emotion analysis engine. For example, if the user is feeling anxious, the engine will select and display a notification message that will give a sense of security. This is the final output on the user's device.

[0982] (Application example 2)

[0983] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0984] In modern logistics centers, inventory management and inspection work remains highly manual and difficult to perform efficiently. Furthermore, there is a lack of support that takes into consideration the emotions of operators, which can lead to excessive stress in the work. Infrastructure inspection work also places a heavy burden on technicians, and it must be performed efficiently and safely. To solve these issues, automation using drones and a system that recognizes users' emotions and provides appropriate feedback are needed.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0986] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying a manager of the generated inspection report, an emotion engine that recognizes a user's emotions, means for displaying information and notifying users based on their emotions, means for a drone to fly over a logistics center area and collect inventory data, and means for processing the collected inventory data and generating an inventory management report. This enables efficient inventory management at logistics centers, efficient inspections of infrastructure structures, and support that takes user emotions into consideration.

[0987] An "aerial vehicle" is an unmanned mechanical device capable of flying in the air, and in this context refers specifically to a drone.

[0988] "Image data" is data that includes visual information acquired by a photographic device such as a drone camera.

[0989] "Sensor data" refers to data about the physical characteristics of the environment and materials collected by various sensors mounted on drones.

[0990] An "image recognition algorithm" is a calculation method for extracting the characteristics of an object from image data and identifying abnormal areas.

[0991] An "inspection report" refers to a report generated based on the results of image recognition algorithms and sensor data analysis.

[0992] The "notification means" refers to a method or device for transmitting the generated inspection report or inventory control report to relevant parties or managers.

[0993] An "emotion engine" is a technology that analyzes a user's facial expressions and voice data to recognize their emotional state.

[0994] "Means for drones to fly over logistics center areas and collect inventory data" refers to a method for drones to fly autonomously over designated areas and obtain information on inventory items.

[0995] An "inventory management report" is a report that lists inventory status and abnormalities generated based on data collected by drones.

[0996] This invention is a system that uses aerial vehicles (drones) to collect and analyze data from infrastructure structures and logistics centers, improving the efficiency of inspections and inventory management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system displays and notifies information that takes the user's emotions into consideration. A specific embodiment of this system is described below.

[0997] Data collection

[0998] The server sends instructions to the drone based on a flight plan sent from the user interface terminal. The drone flies over designated infrastructure structures and logistics center areas, collecting image data using its camera and sensor data using various sensors. The collected data is sent to the server in real time.

[0999] Data analysis

[1000] The server uses image data sent from the drone to run image recognition algorithms and analyze abnormalities in infrastructure and inventory information within the logistics center. To detect abnormalities, it uses pattern recognition rules to identify cracks, rust, corrosion, and other abnormalities. It also analyzes sensor data and evaluates information such as temperature, humidity, seismic intensity, and pressure. Based on the results of this analysis, it automatically generates inspection reports for infrastructure and inventory management reports for the logistics center.

[1001] Reporting and Notifications

[1002] The server automatically generates a report of the analysis results, outputs the report in PDF format, and notifies administrators and other relevant parties by email. If a significant abnormality or inventory abnormality is discovered, an alert is sent immediately.

[1003] User Interface and Emotion Engine

[1004] Using a dedicated user interface terminal, users can specify inspection targets and inventory management areas and check the results in real time. The terminal displays detailed inspection results and inventory status, and past data can also be easily referenced. In addition, an emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling stressed, the server will select a reassuring notification message and display it to the user.

[1005] Specific examples

[1006] For example, when conducting an inventory inspection at a logistics center, the user sets the location information of inventory shelves and an inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the inventory shelves and collecting sensor data. The server receives this data in real time and uses image recognition algorithms to check inventory and detect abnormalities. Based on the detection results, the server automatically generates an inventory management report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately adjusted by the emotion engine.

[1007] Prompt Sentence Examples

[1008] "We will begin an inspection of inventory shelf area A. Please fly the drone into the area and scan the product barcodes."

[1009] "There is an abnormality in the data being analyzed. The upper shelf of inventory shelf A is misaligned and needs to be checked."

[1010] "Stress response detected. Providing advice for efficient stock replenishment."

[1011] This will enable efficient inventory management at logistics centers and inspection of infrastructure structures, as well as support that takes user emotions into consideration.

[1012] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1013] Step 1:

[1014] Users log in using a dedicated terminal and specify the inventory area of ​​the logistics center and the infrastructure structures to be inspected. The input data required is the location information of the target area and a detailed inspection plan. The terminal sends this information to the server. The output is flight plan data.

[1015] Step 2:

[1016] The server sends flight instructions to the drone based on the received flight plan data. The drone then begins flying toward the designated area. The input data is the flight plan data, and the output data is the drone's flight path information.

[1017] Step 3:

[1018] While flying over a designated area, the drone collects image data using a camera and sensor data using various sensors. The input data is the flight path, and the output data is the collected image data and sensor data.

[1019] Step 4:

[1020] The drones send the collected data to a server in real time. The server receives the data and stores it in a database. The input data is image data and sensor data from the drones, and the output data is stored data for analysis.

[1021] Step 5:

[1022] The server runs an image recognition algorithm on the received image data. The algorithm identifies abnormalities such as cracks, rust, and corrosion from the image data. The input data is the image data, and the output data is the analysis results of the abnormalities.

[1023] Step 6:

[1024] The server analyzes the sensor data and obtains information such as temperature, humidity, seismic intensity, and pressure. The input data is the sensor data, and the output data is the analysis result of the environmental information.

[1025] Step 7:

[1026] The server automatically generates inspection reports and inventory management reports based on the analysis results of image data and sensor data. The input data is the analysis results, and the output data is the report.

[1027] Step 8:

[1028] The server creates the generated report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. The input data is the report, and the output data is a notification email.

[1029] Step 9:

[1030] Users can check inspection results and inventory status in real time on a dashboard on a dedicated terminal. The input data is a report from the server, and the output data is the dashboard display content.

[1031] Step 10:

[1032] The emotion engine installed in the dedicated device analyzes the user's facial expressions and voice to recognize their emotional state. The input data is the user's facial expressions and voice data, and the output data is the emotion recognition results.

[1033] Step 11:

[1034] The server selects and displays an appropriate notification message depending on the user's emotional state. For example, if the user is feeling stressed, a message that provides reassurance is displayed. The input data is the emotion recognition result, and the output data is the display message.

[1035] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1036] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1037] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1038] [Fourth embodiment]

[1039] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1040] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1041] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1042] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1043] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1044] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1045] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1046] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1047] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1048] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1049] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1050] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1051] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1052] This invention is a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports, reducing the burden on engineers and enabling efficient and safe infrastructure inspections.

[1053] System Configuration

[1054] The system consists of the following main components:

[1055] 1. Aerial Vehicles

[1056] The drones collect images and sensor data of infrastructure structures as they fly.

[1057] 2. Server

[1058] The server receives, stores, and analyzes the data sent from the drone.

[1059] 3. User Interface Terminal

[1060] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[1061] Program processing explanation

[1062] Data collection

[1063] The user uses a dedicated terminal to set the location information of the infrastructure to be inspected and a flight plan for the drone. Once the settings are complete, the terminal sends these instructions to the server. The server issues instructions to the drone based on the received information, causing the drone to fly automatically along the specified flight route. During flight, the drone uses cameras and various sensors to acquire image data and sensor data of the infrastructure, and sends this to the server in real time.

[1064] Data analysis

[1065] The server runs an image recognition algorithm on the received image data to identify abnormalities such as cracks, rust, and corrosion. The image recognition algorithm uses pre-trained pattern recognition technology to detect abnormalities with high accuracy. It also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity. This allows it to pinpoint abnormalities in detail.

[1066] Reporting and Notifications

[1067] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this inspection report in PDF format and emails it to managers and other relevant parties. It also has an alert function that ensures immediate notification if any important anomalies are discovered.

[1068] User Interface

[1069] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays the location of any abnormalities and detailed analysis results, and past inspection data can also be easily searched and referenced. This enables quick and efficient response.

[1070] Specific examples

[1071] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can then check the report on their device and develop a repair plan as needed.

[1072] This will enable efficient measures to address the deterioration of infrastructure and resolve problems caused by labor shortages.

[1073] The processing flow will be explained below.

[1074] Step 1:

[1075] The user uses a dedicated terminal to specify the infrastructure to be inspected and set up a flight plan, which includes the location of the inspection site, the drone's flight path, and the time of day.

[1076] Step 2:

[1077] The terminal sends the set flight plan to the server, which receives it and sends flight plan instructions to the drone.

[1078] Step 3:

[1079] The drone automatically flies a predetermined route according to a flight plan, and during flight, it uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure.

[1080] Step 4:

[1081] The drones transmit the collected image data and sensor data in real time to a server, which receives the data and stores it in a database.

[1082] Step 5:

[1083] The server then runs image recognition algorithms on the received image data, which automatically detects abnormalities such as cracks, rust, and corrosion.

[1084] Step 6:

[1085] The server performs a detailed analysis of the detected anomalies, identifying their location, range, depth, etc., and generates the analysis results as text data.

[1086] Step 7:

[1087] The server also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, pressure, etc. If an abnormality is detected, it also identifies the location, type, and severity of the abnormality.

[1088] Step 8:

[1089] The server automatically generates an inspection report based on the analysis results, which includes detailed information on identified abnormalities and the priority of areas requiring repair.

[1090] Step 9:

[1091] The server outputs the generated inspection report in PDF format and notifies the administrator and other relevant parties by email. In addition, if any abnormalities are found to be urgent, an alert will be sent immediately.

[1092] Step 10:

[1093] Users can check inspection results in real time through a dashboard on a dedicated device, which displays details of abnormalities and analysis results, and can also easily refer to past inspection data.

[1094] Step 11:

[1095] Users can create repair plans via their terminals, which include the necessary work and schedules.

[1096] Step 12:

[1097] The server stores the details of the repair plan and updates the database so that it can be used at the next inspection.

[1098] This will enable the creation of a system that allows for efficient and safe infrastructure inspection and repair.

[1099] Example 1

[1100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1101] Modern infrastructure structures are diverse, and their inspections require a huge amount of time and effort. Furthermore, traditional inspection methods require engineers to work at height or in dangerous environments, threatening their safety. Furthermore, manual inspections carry a high risk of overlooking or false positives, creating a need for efficient and highly accurate inspections.

[1102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1103] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for transmitting location information and a flight plan of the inspection target from a user terminal, means for executing an image recognition algorithm for identifying abnormalities in the infrastructure based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, and means for displaying the analysis results of the abnormalities to the user in real time. This enables efficient and high-precision inspection of infrastructure, reduces the burden on engineers, and improves safety.

[1104] An "aerial vehicle" is an unmanned flying device, such as a drone. It is primarily used to collect image data and sensor data over a wide area.

[1105] "Image data" refers to photographic images and videos taken by aerial vehicles, and is information used to visually grasp the appearance and condition of infrastructure structures.

[1106] "Sensor data" refers to physical data collected from various sensors (e.g., temperature sensors, humidity sensors, seismic intensity sensors) mounted on airborne vehicles, and is information used to numerically grasp environmental conditions and the status of infrastructure structures.

[1107] "User terminal" refers to an electronic device operated by a user, such as a tablet or laptop computer with dedicated software installed. It is used to set and transmit inspection target location information and flight plans.

[1108] A "server" is a computer that stores and processes data on a network, and receives, analyzes, stores, and reports data sent from airborne vehicles.

[1109] An "image recognition algorithm" is a technology for analyzing image data to detect specific patterns or abnormalities, and includes pattern recognition technology that uses AI and machine learning.

[1110] An "inspection report" is a document that summarizes the inspection results, including any identified abnormalities and the priority of necessary repair work. It is used to report to managers and other relevant parties.

[1111] "Real-time display" means that data is displayed to the user as soon as it is collected, allowing for rapid response in the field.

[1112] "Abnormal areas" refer to problems such as cracks, rust, and corrosion that exist in infrastructure structures, and are areas that require repair or restoration.

[1113] This invention is a system that uses an aerial vehicle to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. This system reduces the burden on engineers and enables efficient and safe infrastructure inspections.

[1114] System Configuration

[1115] The system consists of the following main components:

[1116] 1. Aerial Vehicles

[1117] Drones such as the DJI Phantom 4 and DJI Mavic 2 fly over the infrastructure and collect images and sensor data.

[1118] 2. Server

[1119] The server receives, stores, and analyzes the data, for example, using a cloud computing service such as AWS EC2 or Google Cloud.

[1120] 3. User Interface Terminal

[1121] A terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports. Tablets and laptops are commonly used as terminals.

[1122] Data collection

[1123] The user uses a dedicated device to input the location information of the infrastructure to be inspected and the flight plan into the drone. This is done using a tablet or laptop with a dedicated application installed. Once the settings are complete, the device sends these instructions to a server via Wi-Fi or 4G / 5G networks.

[1124] The server issues instructions to the drone based on the received information, causing it to fly automatically along the designated route. The drone is equipped with a camera and various sensors, and acquires image data and sensor data (temperature, humidity, seismic intensity, etc.) of infrastructure structures during flight, which is then sent to the server in real time.

[1125] Data analysis

[1126] When the server receives the transmitted image data, it analyzes the data using image recognition algorithms such as TensorFlow and OpenCV. The analysis algorithm uses pre-trained pattern recognition rules to identify abnormalities such as cracks, rust, and corrosion. At the same time, it also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, and seismic intensity. This allows for detailed identification of abnormalities.

[1127] Reporting and Notifications

[1128] Once the analysis is complete, the server automatically generates an inspection report using Python or JavaScript. The report contains detailed information about any identified anomalies and the priority of any necessary repair work. The server generates this report in PDF format and emails it to administrators and other relevant parties using the SMTP protocol. If any serious anomalies are discovered, an alert can be sent immediately using the Twilio API or similar.

[1129] User Interface

[1130] Users can view inspection results in real time via a dedicated terminal. The terminal's dashboard displays abnormalities and detailed analysis results. This uses an interactive user interface built using React and Angular.js. Users can also easily search and reference past inspection data, enabling quick and efficient response.

[1131] Specific examples

[1132] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server analyzes the received data in real time and identifies cracks and corrosion. An inspection report is automatically generated based on the detection results and sent to the user's device, where the user can review the report and develop any necessary repair plans.

[1133] An example of a prompt sentence to be input to a specific generative AI model:

[1134] Prepare a bridge inspection report using the following information:

[1135] Name of the bridge: XX Bridge

[1136] Location information:35.710063, 139.8107

[1137] Acquired data: Image data (24 images in total), sensor data (temperature, humidity, seismic intensity)

[1138] Abnormalities: Cracks (5 places), rust (3 places)

[1139] Details of the abnormality: average crack length 0.5m, average rust area 10cm²

[1140] Based on this, generate an inspection report in PDF format.

[1141] The present invention makes it possible to efficiently address the deterioration of infrastructure, thereby resolving labor shortages and safety issues.

[1142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1143] Step 1: Setting up input data

[1144] The user uses a dedicated device to set the location information and flight plan for the infrastructure to be inspected. Input is made using a tablet or laptop, and detailed local coordinate information (latitude and longitude) and flight route are entered through a dedicated application. This generates instructions for the drone to navigate along the flight path.

[1145] Step 2: Submit your flight plan

[1146] The device transmits the location information of the inspection target and the flight plan set by the user to a server via Wi-Fi or 4G / 5G network. The server analyzes the received data and determines the route the drone should take. The generated data is then passed to the drone control API.

[1147] Step 3: Start flying the drone automatically

[1148] The server issues automatic flight instructions to the drone, using the drone's built-in flight control system. As the drone flies automatically along a set route, it uses its camera to capture image data of infrastructure structures and its sensors to collect physical data such as temperature, humidity, and seismic intensity. The collected data is sent to the server in real time for subsequent data analysis processing.

[1149] Step 4: Receiving and storing data

[1150] The server receives the image data and sensor data sent from the drone and first stores it in a database. The received data includes large amounts of image files and sensor output, which are later used for analytical processing. Amazon S3 or Google Cloud Storage are often used for server storage.

[1151] Step 5: Analyze imagery and sensor data

[1152] The server runs image recognition algorithms on the received and stored image data. For example, it uses TensorFlow or OpenCV to identify abnormalities such as cracks, rust, and corrosion. It analyzes the input data (image data) for image recognition processing and generates output results indicating abnormalities. These results are saved in JSON format and used to generate subsequent reports. At the same time, it also analyzes sensor data to evaluate physical characteristics such as temperature, humidity, and seismic intensity.

[1153] Step 6: Generate an inspection report

[1154] The server automatically generates a draft inspection report based on the results of the image and sensor data analysis. It generates the report in PDF format using Python's ReportLab library, among other tools. The report includes detailed information about identified anomalies and the priority of recommended repair work. The report is saved for future notification.

[1155] Step 7: Report Notifications and Alerts

[1156] The server notifies the administrator and relevant parties of the generated inspection report. The report is sent by email using the SMTP protocol. If a serious abnormality is detected, an alert is sent immediately via SMS or phone using the Twilio API. This makes it possible to respond quickly when a rapid response is required.

[1157] Step 8: Viewing in the User Interface

[1158] Users can view inspection results in real time via a dashboard on a dedicated device. The device dashboard provides an interactive interface built using React and Angular.js. Details of abnormalities and analysis results are visually displayed here, and past inspection data can be easily searched and referenced. This enables quick and efficient response.

[1159] (Application example 1)

[1160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1161] Inspecting equipment within a factory requires inspecting high places and complex structural parts, and conventional methods place a heavy burden on personnel and pose safety issues. Furthermore, identifying abnormalities and creating inspection reports was done manually, which was inefficient and prone to errors. For this reason, there was a need for a system that could inspect factory equipment safely and efficiently and identify abnormalities in real time.

[1162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1163] In this invention, the server includes means for receiving image data and sensor data from equipment in the factory, means for executing an image recognition algorithm using AI technology to identify abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities and saving the generated report in a cloud server, and means for notifying an administrator of the generated inspection report in real time. This enables safe and efficient inspection of factory equipment, enables abnormalities to be identified in real time, and enables rapid response.

[1164] "Factory facilities" refers to the infrastructure used in factories, such as various machines, devices, piping, and electrical equipment.

[1165] "Image data" is data representing visual information captured using a camera or other imaging device.

[1166] "Sensor data" refers to data representing physical or environmental information obtained from various sensors.

[1167] "Means for receiving" refers to a combination of hardware and software for remotely obtaining data and processing the obtained data.

[1168] An "image recognition algorithm" is a computational procedure for analyzing image data and detecting specific patterns or anomalies within it.

[1169] "AI technology" is a technology that uses artificial intelligence to analyze data and perform pattern recognition.

[1170] "Automatic generation" refers to the process where a system automatically generates a report based on data without human intervention.

[1171] A "cloud server" is a remote virtual server used to store and process data over the Internet.

[1172] "Means of real-time notification" refers to a method or device that immediately transmits information to managers and relevant parties without delay when an abnormality is detected.

[1173] In this invention, we build a system for efficiently and safely inspecting equipment in factories. The system consists of a drone, a cloud server, and a smartphone app. To implement this invention, the following hardware and software are used.

[1174] Hardware

[1175] Drone: Equipped with a camera, temperature and humidity sensors, seismometer, and pressure sensor.

[1176] Cloud server: A remote server (typically AWS or Google Cloud) for storing and analyzing data.

[1177] User device: Smartphone or tablet (iOS or Android based).

[1178] software

[1179] Drone control software: Used for automated drone flight and data collection (e.g., DJI SDK).

[1180] Server communication library: Sends and receives data from the drone to the cloud server (e.g., HTTP communication library).

[1181] AI analysis library: Analyzes image data and sensor data to identify abnormalities (TensorFlow, OpenCV, etc.).

[1182] Report generation library: Automatically generate inspection reports (e.g., ReportLab).

[1183] Notification system: Provides real-time notifications to user devices (e.g., Firebase Cloud Messaging).

[1184] User interface: The application that runs on a smartphone or tablet.

[1185] System Operation

[1186] 1. Data Collection

[1187] Users use a smartphone app to enter information about the equipment they want to inspect and their inspection plan. Once the settings are complete, the cloud server transmits the information to the drone. The drone then flies automatically and uses its camera and sensors to collect image data and sensor data from the equipment. The data is then sent to the cloud server in real time.

[1188] 2. Data Analysis

[1189] The cloud server then runs an image recognition algorithm using AI technology on the received data. This algorithm detects abnormalities such as cracks, rust, and corrosion, and also identifies abnormalities in high places and complex structural parts. The results of this analysis are then stored on the cloud server.

[1190] 3. Report generation and notifications

[1191] Based on the analysis results, the cloud server automatically generates an inspection report and stores it in the cloud. If any significant abnormalities are detected, a notification system sends an alert to the user's device in real time. Users can check the inspection results on their smartphones or tablets and take any necessary measures.

[1192] Specific examples

[1193] For example, when conducting regular inspections of pipelines within a factory, the user launches the app and inputs the location information of the pipeline to be inspected and the inspection plan. The drone flies automatically according to instructions, capturing sensor data while photographing the area around the pipeline. The cloud server receives this data and performs AI analysis. If cracks, rust, etc. are identified, an inspection report is immediately generated and a notification is sent to the user's device. The user can check the report on the app and create a maintenance plan.

[1194] Prompt Sentence Examples

[1195] "We are thinking about a system that uses drones to detect abnormalities in factory equipment. In this system, drones fly automatically to collect image data and sensor data from the equipment, which is then analyzed by AI to identify any abnormalities. Specifically, please explain in detail what steps should be taken to build the system, including the hardware and software to be used."

[1196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1197] Step 1:

[1198] The user launches the smartphone app and inputs information about the equipment they want to inspect (for example, the equipment's location and inspection plan). This information is used as input. The smartphone app then sends this input data to a cloud server. The data sent to the server becomes the output for creating an automatic flight plan for the drone.

[1199] Step 2:

[1200] The cloud server issues automatic flight instructions to the drone based on the received inspection plan. Once the flight plan is sent to the drone, the drone automatically flies along the designated route, collecting image data using its camera and sensor data using various sensors (temperature and humidity sensors, seismometers, pressure sensors). This becomes input data. The collected image data and sensor data are sent to the cloud server in real time and become output for analysis.

[1201] Step 3:

[1202] The cloud server runs an image recognition algorithm using AI technology on the image data and sensor data it receives in real time. Image data and sensor data are used as input data. The cloud server analyzes the image data using AI technology (for example, TensorFlow or OpenCV) to detect abnormalities such as cracks, rust, and corrosion. Detailed data on detected abnormalities is output.

[1203] Step 4:

[1204] The cloud server automatically generates an inspection report based on details of abnormalities detected by the image recognition algorithm. Detailed data on the abnormalities is used as input data. The server converts the generated report into PDF format and saves it in the cloud. The generated inspection report is the output.

[1205] Step 5:

[1206] The cloud server prepares the generated inspection report for notification to the administrator. The generated inspection report is used as input data. The server notifies the administrator via a notification system (e.g., Firebase Cloud Messaging) that the report is available to their smartphone or tablet. This is the output. If any significant anomalies are detected, an alert is sent immediately.

[1207] Step 6:

[1208] Users can check the inspection report through the application on their smartphone or tablet. The inspection report received after receiving the notification is used as input data. Users can check detailed abnormalities and inspection results through the app interface and plan necessary countermeasures. The checked report contents are output.

[1209] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1210] This invention is a system that uses an aerial vehicle (drone) to collect and analyze image data and sensor data of infrastructure structures, generates inspection reports, and combines this with an emotion engine that recognizes the user's emotions. This system reduces the burden on engineers, enables efficient and safe infrastructure inspections, and presents information that takes user emotions into consideration.

[1211] System Configuration

[1212] The system consists of the following main components:

[1213] 1. Aerial Vehicles

[1214] The drones collect images and sensor data of infrastructure structures as they fly.

[1215] 2. Server

[1216] The server receives, stores, and analyzes the data sent from the drone.

[1217] 3. User Interface Terminal

[1218] The terminal operated by the user provides an interface for issuing inspection instructions and viewing analysis results and inspection reports.

[1219] 4. Emotion Engine

[1220] The emotion engine recognizes the user's emotions by analyzing their facial expressions and voice, and then displays appropriate information and notifications accordingly.

[1221] Program processing explanation

[1222] Data collection

[1223] The user uses a dedicated terminal to specify the infrastructure inspection target and set up a flight plan. The settings include the location information of the inspection location, the drone's flight path, and the time period. The terminal sends the set flight plan to the server. The server receives this and sends flight plan instructions to the drone. The drone follows the instructed flight plan and automatically flies the predetermined flight path. During flight, the drone uses cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure. The drone sends the collected image data and sensor data to the server in real time.

[1224] Data analysis

[1225] The server runs an image recognition algorithm on the received image data. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities, identifying the location, range, depth, etc. of the abnormality and generating the analysis results as text data. It also analyzes sensor data and evaluates physical characteristics such as temperature, humidity, seismic intensity, and pressure. If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[1226] Report Generation

[1227] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and other relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[1228] User Interface and Emotion Engine

[1229] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[1230] Specific examples

[1231] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[1232] This will enable efficient measures to deal with aging infrastructure and enable system operations that take user feelings into consideration.

[1233] The processing flow will be explained below.

[1234] Step 1:

[1235] Using a dedicated terminal, the user selects the infrastructure to be inspected and sets up a flight plan, including details such as the target location, flight path, shooting interval, type of sensor to be used, and flight time.

[1236] Step 2:

[1237] The terminal sends the configured flight plan to the server, which receives the flight plan and transmits it to the drone.

[1238] Step 3:

[1239] The drone will automatically begin flying according to the received flight plan, and during the flight, it will use its cameras and sensors to collect image and sensor data.

[1240] Step 4:

[1241] The drone transmits the collected image data and sensor data to a server in real time.

[1242] Step 5:

[1243] The server runs an image recognition algorithm on the received image data, which uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[1244] Step 6:

[1245] The server identifies any abnormalities detected through image analysis and converts their location, range, and depth into text data as the analysis results.

[1246] Step 7:

[1247] The server also analyzes sensor data, such as seismic intensity, temperature, humidity, and pressure, and if an abnormality is detected, identifies the type and severity of the abnormality.

[1248] Step 8:

[1249] The server automatically generates an inspection report based on the analysis results, which includes text and diagrams and includes detailed information about any abnormalities and the priority of repairs.

[1250] Step 9:

[1251] The server notifies the administrator of the generated inspection report via email or SMS, with the report itself attached in PDF format. It also sends an immediate alert for any abnormalities that are deemed urgent.

[1252] Step 10:

[1253] Users can check the inspection and analysis results in real time through a dashboard on a dedicated device, which also displays detailed information on abnormalities and past inspection data.

[1254] Step 11:

[1255] The emotion engine recognizes the user's emotional state as they use the dashboard. The emotion engine analyzes the user's facial expressions and voice in real time to identify the user's emotional state.

[1256] Step 12:

[1257] The device adjusts the content and display of notification messages received by the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the device will display a notification message that gives a sense of security.

[1258] Step 13:

[1259] Users can review coordinated notifications and inspection reports and create necessary repair plans, which include specific work content and schedules.

[1260] Step 14:

[1261] The server stores the proposed repair plan in a database and updates the data so that it can be used at the next inspection.

[1262] This will enable efficient and safe infrastructure inspection and repair, while also providing information that takes user feelings into consideration.

[1263] Example 2

[1264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1265] In conventional infrastructure inspection systems, the analysis and reporting of inspection results was done manually, which took time and effort and could lead to variations in the accuracy of the analysis. This also placed a heavy burden on engineers, posing safety and efficiency challenges. Furthermore, the system did not present information that took user emotions into consideration, potentially increasing user anxiety and stress.

[1266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1267] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying an administrator of the generated inspection report, emotion analysis means for recognizing the user's emotional state and adjusting the notification message, and user interface means for displaying data in real time. This enables infrastructure inspection work to be performed efficiently and safely, reducing the burden on engineers. Furthermore, since information is presented taking the user's emotions into consideration, it is possible to reduce user anxiety and stress.

[1268] "Aerial vehicle" is a general term for unmanned aerial vehicles that fly over infrastructure structures such as stores and facilities to collect data.

[1269] "Image data" refers to visual information of infrastructure structures photographed by a camera mounted on an aerial vehicle.

[1270] "Sensor data" refers to physical property data such as temperature, humidity, pressure, and seismic intensity measured by various sensors mounted on airborne vehicles.

[1271] "Means for receiving" means the equipment or software functions for receiving and storing image data and sensor data transmitted from an airborne vehicle.

[1272] An "image recognition algorithm" is a program or method for automatically identifying abnormalities such as cracks, rust, and corrosion in received image data.

[1273] An "inspection report" is an automatically generated report based on abnormalities identified by an image recognition algorithm, and includes detailed information about the abnormalities and the priority of repair work.

[1274] The "means for notifying the administrator" refers to a communication means or software function for notifying the administrator or related parties of the generated inspection report in real time.

[1275] The "emotion analysis means" is a device or software function for analyzing the user's facial expressions and voice and recognizing the user's emotions.

[1276] "User interface means" refers to the function of a device or software that provides an interface that allows users to view and manipulate inspection data and results in real time.

[1277] This invention relates to a system that uses aerial vehicles (drones) to collect and analyze image data and sensor data of infrastructure structures and generate inspection reports. Furthermore, by combining this with an emotion engine that recognizes user emotions and presents appropriate information, it is possible to reduce the burden on engineers and perform infrastructure inspections efficiently and safely.

[1278] To implement this system, the following major hardware and software are used:

[1279] Hardware

[1280] 1. Aerial vehicle (drone): Equipped with a camera and multiple sensors (temperature and humidity sensors, pressure sensors, etc.), it has the ability to collect image data and sensor data.

[1281] 2. Server: A high-performance computing system for receiving, storing, and analyzing data sent from the drones.

[1282] 3. User interface terminal: A device operated by the user that provides an interface for issuing inspection instructions and checking analysis results and inspection reports.

[1283] software

[1284] 1. Image recognition algorithm: An image analysis program for detecting abnormalities such as cracks, rust, and corrosion. It is composed mainly of deep learning models.

[1285] 2. Data analysis software: A program that analyzes sensor data (temperature, humidity, pressure, seismic intensity, etc.) and evaluates physical properties.

[1286] 3. Emotion analysis engine: Software that analyzes the user's facial expressions and voice to recognize their emotional state.

[1287] 4. User interface software: An interface program designed to display data in real time and allow users to easily operate it.

[1288] Specific actions

[1289] The user uses a dedicated device to specify the infrastructure to be inspected and set up a flight plan, including the location of the inspection site, the drone's flight path, and the time of day. The flight path includes detailed instructions, such as covering both the top and bottom of a bridge.

[1290] The terminal sends the set flight plan to the server. The server receives it and sends flight plan instructions to the drone. The drone follows the instructed flight plan, automatically flying along a predetermined flight path while using cameras and sensors to collect image data and sensor data such as temperature, humidity, and pressure, and sends this data to the server in real time.

[1291] The server stores the received data in a database and runs an image recognition algorithm on the image data. This algorithm automatically detects abnormalities such as cracks, rust, and corrosion. It then performs a detailed analysis of the detected abnormalities to identify their location, range, depth, etc. It also analyzes sensor data to evaluate physical characteristics (temperature, humidity, seismic intensity, pressure, etc.). If an abnormality is detected, it also identifies the location, type, and degree of the abnormality.

[1292] Once the analysis is complete, the server automatically generates a draft inspection report, which includes detailed information about any identified anomalies and the priority of any necessary repair work. The server outputs the generated inspection report in PDF format and notifies administrators and relevant parties via email. It also immediately sends alerts if any critical anomalies are discovered.

[1293] Users can check the inspection results in real time through a dashboard on a dedicated device. The dashboard displays the locations of any abnormalities discovered and detailed analysis results, and allows easy reference to past inspection data. The emotion engine also analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will use that information to select a notification message that provides a sense of security.

[1294] Specific examples

[1295] For example, when conducting regular inspections of a bridge, the user sets the bridge's location information and inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the area around the bridge and collecting sensor data. The server receives this data in real time and uses an image recognition algorithm to detect cracks. Based on the detection results, the server automatically generates an inspection report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately tailored by the emotion engine. For example, even if urgent repairs are required, a thoughtful message is displayed to prevent the user from feeling overly anxious.

[1296] Prompt Sentence Examples

[1297] 1. Drone flight plan configuration: "Please explain the process for configuring infrastructure location information and inspection plans (flight path, time slots, etc.) and sending them to the server."

[1298] 2. Data Analysis: Please explain in detail how the server analyzes the received image data and sensor data to detect any abnormalities.

[1299] 3. Report Generation: "Please explain in detail the steps by which the server automatically generates an inspection report based on the analysis results."

[1300] 4. How the Emotion Engine Works: "Please explain how the emotion engine works, recognizing the user's emotions and displaying appropriate information or notifications."

[1301] This will enable efficient measures to deal with aging infrastructure, and will also enable system operations that take user feelings into consideration.

[1302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1303] Step 1:

[1304] The user uses a dedicated terminal to specify the infrastructure structure (e.g., a bridge) to be inspected. Input includes the location information of the inspection location, the drone's flight path, and the flight time. The terminal compiles this information into a drone flight plan and generates data in JSON format. This data becomes the terminal's output.

[1305] Step 2:

[1306] The device sends the generated flight plan data to the server. The server receives this data and generates instruction data for the drone based on it. The instruction data includes the specific flight path, subject to be photographed, and timing of the photograph. The server's output is instruction data for the drone.

[1307] Step 3:

[1308] The server sends instruction data to the drone. Based on the received instruction data, the drone automatically begins flying according to the set flight path. During flight, the drone uses its on-board camera and sensors to collect image data and sensor data. This is the drone's output data.

[1309] Step 4:

[1310] The drone transmits the collected image data and sensor data to the server in real time. The server receives this data and stores it in a database. The stored image data and sensor data become the input for the server.

[1311] Step 5:

[1312] The server receives the stored image data as input and runs an image recognition algorithm. The algorithm automatically detects abnormalities such as cracks, rust, and corrosion. Information about the detected abnormalities (location, range, depth, etc.) is output from the server. The server also analyzes sensor data and evaluates physical properties (temperature, humidity, seismic intensity, pressure, etc.). The results of these analyses are also output from the server.

[1313] Step 6:

[1314] The server automatically generates an inspection report based on the results of the image recognition algorithm and sensor data analysis. The report contains detailed information about any identified abnormalities and the priority of repair work. The generated inspection report is the output of the server.

[1315] Step 7:

[1316] The server outputs the generated inspection report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. This is the output of the server.

[1317] Step 8:

[1318] The generated inspection report is displayed on the dashboard of the user's dedicated terminal. Through the dashboard, the user can check the report contents and analysis results in real time. The dashboard display is the output of the user's terminal.

[1319] Step 9:

[1320] The emotion analysis engine analyzes the face and voice of the user using the dashboard to recognize the user's emotional state. The analysis results are the output of the emotion analysis engine. For example, if the user is feeling anxious, the engine will select and display a notification message that will give a sense of security. This is the final output on the user's device.

[1321] (Application example 2)

[1322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1323] In modern logistics centers, inventory management and inspection work remains highly manual and difficult to perform efficiently. Furthermore, there is a lack of support that takes into consideration the emotions of operators, which can lead to excessive stress in the work. Infrastructure inspection work also places a heavy burden on technicians, and it must be performed efficiently and safely. To solve these issues, automation using drones and a system that recognizes users' emotions and provides appropriate feedback are needed.

[1324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1325] In this invention, the server includes means for receiving image data and sensor data from the aerial vehicle, means for executing an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data, means for automatically generating an inspection report based on the identified abnormalities, means for notifying a manager of the generated inspection report, an emotion engine that recognizes a user's emotions, means for displaying information and notifying users based on their emotions, means for a drone to fly over a logistics center area and collect inventory data, and means for processing the collected inventory data and generating an inventory management report. This enables efficient inventory management at logistics centers, efficient inspections of infrastructure structures, and support that takes user emotions into consideration.

[1326] An "aerial vehicle" is an unmanned mechanical device capable of flying in the air, and in this context refers specifically to a drone.

[1327] "Image data" is data that includes visual information acquired by a photographic device such as a drone camera.

[1328] "Sensor data" refers to data about the physical characteristics of the environment and materials collected by various sensors mounted on drones.

[1329] An "image recognition algorithm" is a calculation method for extracting the characteristics of an object from image data and identifying abnormal areas.

[1330] An "inspection report" refers to a report generated based on the results of image recognition algorithms and sensor data analysis.

[1331] The "notification means" refers to a method or device for transmitting the generated inspection report or inventory control report to relevant parties or managers.

[1332] An "emotion engine" is a technology that analyzes a user's facial expressions and voice data to recognize their emotional state.

[1333] "Means for drones to fly over logistics center areas and collect inventory data" refers to a method for drones to fly autonomously over designated areas and obtain information on inventory items.

[1334] An "inventory management report" is a report that lists inventory status and abnormalities generated based on data collected by drones.

[1335] This invention is a system that uses aerial vehicles (drones) to collect and analyze data from infrastructure structures and logistics centers, improving the efficiency of inspections and inventory management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system displays and notifies information that takes the user's emotions into consideration. A specific embodiment of this system is described below.

[1336] Data collection

[1337] The server sends instructions to the drone based on a flight plan sent from the user interface terminal. The drone flies over designated infrastructure structures and logistics center areas, collecting image data using its camera and sensor data using various sensors. The collected data is sent to the server in real time.

[1338] Data analysis

[1339] The server uses image data sent from the drone to run image recognition algorithms and analyze abnormalities in infrastructure and inventory information within the logistics center. To detect abnormalities, it uses pattern recognition rules to identify cracks, rust, corrosion, and other abnormalities. It also analyzes sensor data and evaluates information such as temperature, humidity, seismic intensity, and pressure. Based on the results of this analysis, it automatically generates inspection reports for infrastructure and inventory management reports for the logistics center.

[1340] Reporting and Notifications

[1341] The server automatically generates a report of the analysis results, outputs the report in PDF format, and notifies administrators and other relevant parties by email. If a significant abnormality or inventory abnormality is discovered, an alert is sent immediately.

[1342] User Interface and Emotion Engine

[1343] Using a dedicated user interface terminal, users can specify inspection targets and inventory management areas and check the results in real time. The terminal displays detailed inspection results and inventory status, and past data can also be easily referenced. In addition, an emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, if the user is feeling stressed, the server will select a reassuring notification message and display it to the user.

[1344] Specific examples

[1345] For example, when conducting an inventory inspection at a logistics center, the user sets the location information of inventory shelves and an inspection plan on a dedicated device. The drone follows the instructed flight path, photographing the inventory shelves and collecting sensor data. The server receives this data in real time and uses image recognition algorithms to check inventory and detect abnormalities. Based on the detection results, the server automatically generates an inventory management report and notifies the user's device. The user can check the report through the device's dashboard and receive notification messages appropriately adjusted by the emotion engine.

[1346] Prompt Sentence Examples

[1347] "We will begin an inspection of inventory shelf area A. Please fly the drone into the area and scan the product barcodes."

[1348] "There is an abnormality in the data being analyzed. The upper shelf of inventory shelf A is misaligned and needs to be checked."

[1349] "Stress response detected. Providing advice for efficient stock replenishment."

[1350] This will enable efficient inventory management at logistics centers and inspection of infrastructure structures, as well as support that takes user emotions into consideration.

[1351] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1352] Step 1:

[1353] Users log in using a dedicated terminal and specify the inventory area of ​​the logistics center and the infrastructure structures to be inspected. The input data required is the location information of the target area and a detailed inspection plan. The terminal sends this information to the server. The output is flight plan data.

[1354] Step 2:

[1355] The server sends flight instructions to the drone based on the received flight plan data. The drone then begins flying toward the designated area. The input data is the flight plan data, and the output data is the drone's flight path information.

[1356] Step 3:

[1357] While flying over a designated area, the drone collects image data using a camera and sensor data using various sensors. The input data is the flight path, and the output data is the collected image data and sensor data.

[1358] Step 4:

[1359] The drones send the collected data to a server in real time. The server receives the data and stores it in a database. The input data is image data and sensor data from the drones, and the output data is stored data for analysis.

[1360] Step 5:

[1361] The server runs an image recognition algorithm on the received image data. The algorithm identifies abnormalities such as cracks, rust, and corrosion from the image data. The input data is the image data, and the output data is the analysis results of the abnormalities.

[1362] Step 6:

[1363] The server analyzes the sensor data and obtains information such as temperature, humidity, seismic intensity, and pressure. The input data is the sensor data, and the output data is the analysis result of the environmental information.

[1364] Step 7:

[1365] The server automatically generates inspection reports and inventory management reports based on the analysis results of image data and sensor data. The input data is the analysis results, and the output data is the report.

[1366] Step 8:

[1367] The server creates the generated report in PDF format and notifies the administrator and related parties by email. If a significant abnormality is detected, an alert is sent immediately. The input data is the report, and the output data is a notification email.

[1368] Step 9:

[1369] Users can check inspection results and inventory status in real time on a dashboard on a dedicated terminal. The input data is a report from the server, and the output data is the dashboard display content.

[1370] Step 10:

[1371] The emotion engine installed in the dedicated device analyzes the user's facial expressions and voice to recognize their emotional state. The input data is the user's facial expressions and voice data, and the output data is the emotion recognition results.

[1372] Step 11:

[1373] The server selects and displays an appropriate notification message depending on the user's emotional state. For example, if the user is feeling stressed, a message that provides reassurance is displayed. The input data is the emotion recognition result, and the output data is the display message.

[1374] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1376] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1377] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1378] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1379] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1380] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1381] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1382] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1383] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1384] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1385] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1386] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1387] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1388] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1389] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1390] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1391] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1392] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1393] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1394] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1395] The following is further disclosed regarding the above embodiment.

[1396] (Claim 1)

[1397] means for receiving image data and sensor data from the airborne vehicle;

[1398] means for executing an image recognition algorithm to identify abnormalities in the infrastructure based on the received data;

[1399] A means for automatically generating an inspection report based on the identified abnormality location;

[1400] a means for notifying an administrator of the generated inspection report;

[1401] A system including:

[1402] (Claim 2)

[1403] 10. The system of claim 1, wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[1404] (Claim 3)

[1405] 2. The system of claim 1, wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure.

[1406] "Example 1"

[1407] (Claim 1)

[1408] means for receiving image data and sensor data from the airborne vehicle;

[1409] A means for transmitting location information and a flight plan of an inspection target from a user terminal;

[1410] means for executing an image recognition algorithm to identify abnormalities in the infrastructure based on the received data;

[1411] A means for automatically generating an inspection report based on the identified abnormality location;

[1412] a means for notifying an administrator of the generated inspection report;

[1413] A means for displaying the analysis results of the abnormality location to the user in real time;

[1414] A system including:

[1415] (Claim 2)

[1416] 10. The system of claim 1, wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[1417] (Claim 3)

[1418] 2. The system of claim 1, wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure.

[1419] "Application Example 1"

[1420] (Claim 1)

[1421] means for receiving image data and sensor data from equipment in a factory;

[1422] A means for using AI technology to execute an image recognition algorithm that identifies abnormalities in infrastructure structures based on the received data;

[1423] a means for automatically generating an inspection report based on the identified abnormality location and storing the generated report in a cloud server;

[1424] A means for notifying the manager of the generated inspection report in real time;

[1425] A system including:

[1426] (Claim 2)

[1427] 10. The system of claim 1, wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion, and has the ability to detect high-altitude and complex structural parts.

[1428] (Claim 3)

[1429] The system according to claim 1, wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure, and the collected data is used for analysis via a cloud server.

[1430] "Example 2: Combining Emotion Engines"

[1431] (Claim 1)

[1432] means for receiving image data and sensor data from the airborne vehicle;

[1433] means for executing an image recognition algorithm to identify abnormalities in the infrastructure based on the received data;

[1434] A means for automatically generating an inspection report based on the identified abnormality location;

[1435] a means for notifying an administrator of the generated inspection report;

[1436] emotion analysis means for recognizing the emotional state of a user and adjusting the notification message;

[1437] a user interface means for displaying data in real time;

[1438] A system including:

[1439] (Claim 2)

[1440] 10. The system of claim 1, wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[1441] (Claim 3)

[1442] 2. The system of claim 1, wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure.

[1443] "Application example 2 when combining emotion engines"

[1444] (Claim 1)

[1445] means for receiving image data and sensor data from the airborne vehicle;

[1446] means for executing an image recognition algorithm to identify abnormalities in the infrastructure based on the received data;

[1447] A means for automatically generating an inspection report based on the identified abnormality location;

[1448] a means for notifying an administrator of the generated inspection report;

[1449] an emotion engine that recognizes the user's emotions;

[1450] A means for displaying information or making a notification according to the user's emotions;

[1451] A means for drones to fly over the distribution center area and collect inventory data;

[1452] means for processing the collected inventory data and generating inventory control reports;

[1453] A system including:

[1454] (Claim 2)

[1455] 10. The system of claim 1, wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

[1456] (Claim 3)

[1457] 2. The system of claim 1, wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure. [Explanation of symbols]

[1458] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving image data and sensor data from the airborne vehicle; means for executing an image recognition algorithm to identify abnormalities in the infrastructure based on the received data; A means for automatically generating an inspection report based on the identified abnormality location; a means for notifying an administrator of the generated inspection report; A system including:

2. The system of claim 1 , wherein the image recognition algorithm uses predefined pattern recognition rules to detect anomalies such as cracks, rust, and corrosion.

3. The system of claim 1 , wherein the sensors include sensors for measuring seismic intensity, temperature, humidity, and pressure.

Images