system

Abstract

We provide the system. [Solution] A system comprising: means for using an unmanned aerial vehicle to patrol and collect data by scanning images of communication equipment installed at high altitudes; means for analyzing the data and detecting anomalies using artificial intelligence; and means for generating optimal maintenance procedures based on the analysis results and presenting them to the worker. The unmanned aerial vehicle (UAV) creates a flight plan in advance from weather and terrain data, patrols the area around the communication equipment, acquires video and environmental data, and transmits it to the server. The server analyzes the video and environmental data, detects anomalies, identifies the anomalies and generates diagnostic results, and transmits the analysis results and maintenance actions to the worker's terminal. The worker's terminal displays the results using augmented reality.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Maintenance work of wireless communication devices is often located at high places, and there is a problem that it is difficult to ensure safety for workers. In addition, the conventional maintenance work largely depends on the experience and knowledge of workers, and it often takes time to discover problems and optimize repair procedures. Furthermore, since the accompanying reduction in work efficiency may affect the reliability of the communication infrastructure, there is a demand for establishing a rapid and accurate maintenance method.

Means for Solving the Problems

[0005] This invention provides a system that reduces worker risk by using an unmanned aerial vehicle to inspect communication equipment installed at high altitudes and collect data from video and sensors. The collected data is analyzed in real time using an artificial intelligence analysis method to automatically detect the presence or absence of abnormalities. Furthermore, based on the information obtained from the analysis, an optimal maintenance procedure is generated considering past maintenance history and presented to the worker, enabling efficient and safe maintenance of the communication equipment. This makes it possible to build a system in which anyone can perform maintenance quickly and accurately, regardless of the worker's experience.

[0006] An "unmanned aerial vehicle" is a device that can fly remotely or autonomously to replace human work in high-altitude or dangerous locations.

[0007] A "communication device" is a device installed to send and receive information such as voice, data, and video.

[0008] "Patrolling" refers to the act of an unmanned aerial vehicle (UAV) flying along a specific route to monitor the surrounding environment.

[0009] "Data" refers to information obtained from images and sensors collected by unmanned aerial vehicles.

[0010] Artificial intelligence is a technology that imitates human intellectual behavior, analyzes data, and performs pattern recognition and decision-making.

[0011] "Analysis" is the process of processing collected data to detect anomalies and identify problems.

[0012] "Maintenance procedures" refer to a series of work steps and methods for properly operating communication equipment and preventing malfunctions.

[0013] A "worker" refers to a technician who is responsible for the practical work of inspecting and repairing communication equipment. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of the present invention includes an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by operators. The roles and operations of each component will be described in detail below.

[0036] The server creates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This flight plan includes route information designed to allow the UAV to patrol the destination safely and efficiently, and also takes weather and terrain data into consideration. Based on this information, the server sends real-time instructions to the UAV.

[0037] The unmanned aerial vehicle (UAV) flies autonomously based on instructions from the server, patrolling the area around the communication device. During this time, it collects video and environmental data from the communication device using its equipped cameras and sensors. This collected data is transmitted to the server in real time.

[0038] The terminal is a device held by the worker and functions as an interface that displays analysis results and maintenance procedures from the server. The server analyzes the transmitted data using artificial intelligence to detect anomalies and identify their causes. The analysis results, along with the optimal maintenance procedures, are sent to the terminal, and the worker uses this information to repair or adjust the communication equipment.

[0039] As a concrete example, if a malfunction occurs in a communication device installed at a high altitude, the unmanned aerial vehicle (UAV) will patrol the area and send detailed video footage back to the server. Based on the video, the server will identify the abnormality, such as a damaged cable or a loose connection, and create specific procedures to correct it. Workers can then safely and efficiently perform repairs by checking the graphical procedures displayed using AR technology through a terminal.

[0040] This system enables the maintenance of communication equipment quickly and accurately, regardless of the operator's experience. The introduction of this system will enhance the reliability of the communication infrastructure and reduce the workload on technicians.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] Based on the location information of communication devices requiring patrol, the server generates a flight plan for the unmanned aerial vehicle (UAV), taking into account weather data and current operational conditions. This flight plan includes details of an efficient and safe patrol route.

[0044] Step 2:

[0045] The unmanned aerial vehicle (UAV) begins flight autonomously according to the flight plan received from the server via the terminal. During flight, it collects video data of the surroundings of the communication device using high-precision cameras and various sensors.

[0046] Step 3:

[0047] The server receives data transmitted in real time from the unmanned aerial vehicle. The server's artificial intelligence analysis engine processes this data in real time and uses image analysis algorithms to detect anomalies in the communication equipment.

[0048] Step 4:

[0049] Based on the detected anomaly, the server generates the optimal repair procedure by comparing it with past repair history. This repair procedure clearly specifies the particular work methods and tools to be used.

[0050] Step 5:

[0051] Repair instructions generated from the server are sent to the user's (worker's) terminal. The terminal displays the maintenance instructions and necessary information visually using augmented reality technology.

[0052] Step 6:

[0053] The user, acting as the operator, utilizes the information and AR guidance provided on the terminal to efficiently and safely repair and adjust communication equipment. This allows the operator to quickly resolve any problems.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Communication equipment installed at high altitudes is difficult to inspect and maintain, requiring early detection and rapid response to any abnormalities. However, conventional methods are dependent on the skill level of the workers and weather conditions, making efficient maintenance challenging. Furthermore, there is a need for automated abnormality detection and the provision of efficient maintenance procedures.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for patrolling communication equipment using a data processing device and collecting information, evaluation means for analyzing the collected information using a machine learning model and detecting anomalies, means for generating and providing optimal maintenance actions to the operator based on the analysis results, and means for creating a flight plan using a machine learning model. This enables early detection of anomalies and the provision of efficient maintenance procedures, independent of the individual capabilities of the operator.

[0059] A "data processing device" is an autonomous, mobile device designed to patrol communication equipment installed at high altitudes and collect information about the surrounding environment.

[0060] "Information" refers to data about the surrounding environment and state of communication equipment, including images and audio acquired through cameras and sensors, as well as environmental measurements.

[0061] A "machine learning model" is a set of algorithms used to analyze large amounts of data, identify patterns, and detect anomalies, and includes artificial intelligence technology.

[0062] "Evaluation means" refers to a system process that uses a machine learning model based on collected information to determine the normality of communication equipment and identify abnormalities.

[0063] "Maintenance actions" refer to an action plan that outlines the work and processes necessary to correct malfunctions in communication equipment, and includes specific instructions for workers.

[0064] A "flight plan" is a plan that outlines the route information a data processing unit will take when patrolling communication equipment, and is designed to maximize safety and efficiency.

[0065] "Means to be provided to the worker" refers to technologies for appropriately presenting the analyzed results and maintenance actions to the worker, and includes visual display methods.

[0066] The embodiments for carrying out the present invention are described below.

[0067] This system includes a data processing unit (unmanned aerial vehicle), a computer system (server) for controlling it and analyzing data, and an operating device (terminal) used by operators. This allows for the rapid detection of abnormalities in communication equipment and enables appropriate maintenance.

[0068] The server generates a flight plan for the data processing unit to patrol the communication equipment. Specifically, the server uses a generation AI model to analyze real-time weather data and terrain information to design a safe and efficient flight path. Furthermore, the server sends commands to the data processing unit via a communication protocol. The MQTT protocol is used as needed.

[0069] The data processing unit autonomously patrols the area around the communication equipment according to instructions from the server, and returns information collected by its equipped cameras and sensors to the server in real time. This information includes image data, environmental data, and acoustic data.

[0070] The server uses the received information to detect anomalies with a machine learning model and generates maintenance actions based on the results. The evaluation method utilizes algorithms suitable for image analysis, such as convolutional neural networks (CNNs).

[0071] Subsequently, the server transmits the analyzed anomaly information and maintenance procedures to the terminal, which then visually presents this information to the worker. Specifically, augmented reality technology is used to display instructions in a format that is easy for the worker to understand.

[0072] For example, if an anomaly is detected in communication equipment installed at a high altitude, a data processing unit flies over the area and collects detailed video footage using a camera. The server analyzes this data, and if an anomaly is identified, it provides the worker with the most suitable repair procedure via a terminal.

[0073] An example of a prompt message for a generative AI model is, "Prepare a means to patrol communication devices and detect anomalies from the collected information."

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The server generates flight plans using a generative AI model. It receives real-time weather data and terrain information as input. This data is analyzed to calculate a safe and efficient route. As output, it creates a flight plan including route information and sends it to a data processing unit. Specifically, the server obtains wind speed and rainfall information via a weather API and performs route optimization based on this information.

[0077] Step 2:

[0078] The data processing unit patrols the area around the communication equipment according to the flight plan received from the server. It receives command routes from the server as input and transmits video and environmental data from the communication equipment to the server as output. Specific operations include acquiring detailed images and measurement data in real time during patrols using cameras and sensors.

[0079] Step 3:

[0080] The server analyzes video and environmental data received from the data processing unit. It receives data from multiple sensors as input and uses a machine learning model to detect anomalies. As output, it generates anomaly identification and diagnostic results, and plans maintenance actions based on these results. Specifically, the server utilizes a convolutional neural network (CNN) to identify anomalies in the video data.

[0081] Step 4:

[0082] The server transmits analysis results and maintenance actions to the terminal. Using the diagnostic results from the analysis as input, it generates specific instructions for the worker as output. These specific actions include creating documents containing repair procedures and important points to note.

[0083] Step 5:

[0084] The terminal provides information to the worker based on instructions from the server. It receives maintenance instructions from the server as input and displays the information in a visually understandable format as output. Specifically, it utilizes augmented reality (AR) technology to display repair locations as 3D models, thereby supporting the worker's understanding.

[0085] Step 6:

[0086] The user (worker) performs the actual maintenance work based on the information displayed on the terminal. The input is the instruction information from the terminal, and the output is the restoration of the communication equipment to normal operation. Specific actions include following the steps indicated by the terminal in order and reporting the completion of the work to the server upon completion.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Traditional inspection and maintenance work on communication equipment involved working at heights, posing safety challenges and placing a heavy burden on workers. Furthermore, detecting anomalies and determining repair methods relied on the worker's experience, resulting in a lack of efficiency and consistency. This made rapid response difficult and posed a risk of compromising the reliability of the communication infrastructure.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to workers based on the analysis results; and means for immediately notifying workers when an anomaly is detected and visually displaying repair and verification procedures using augmented reality technology. This enables safe and efficient patrolling and maintenance of communication equipment, reduces the burden on workers, and improves the reliability of the communication infrastructure.

[0092] An "unmanned aerial vehicle" is a device that flies remotely or autonomously and performs a specific function.

[0093] A "communication device" is an electronic device installed at a high location that transmits and receives data.

[0094] "Data" refers to information about the communication equipment and surrounding environment collected by the unmanned aerial vehicle.

[0095] "Artificial intelligence" is a technology in which computer programs mimic human intelligence, analyze data, and detect anomalies.

[0096] "Analysis means" refers to methods or mechanisms for evaluating data and determining whether or not there are abnormalities.

[0097] A "maintenance procedure" is a standard set of operations established to correct malfunctions in communication equipment.

[0098] Augmented reality technology is a technology that overlays virtual information onto the real world's field of view.

[0099] A "smartphone" is a portable information processing device that allows for communication and the use of various applications.

[0100] A "head-mounted display" is a device that displays images in the wearer's field of vision.

[0101] "Notification" refers to the act of communicating information to inform workers of the occurrence of an abnormality.

[0102] The present invention comprises an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by an operator. The following method is used to realize this system.

[0103] The server generates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. The server considers weather data and terrain information to create route information that enables autonomous flight, and sends real-time instructions to the UAV. In this process, GIS data is used to design a safe and efficient route.

[0104] The unmanned aerial vehicle (UAV) uses its onboard cameras and sensors to collect video data from communication devices and data about the surrounding environment. The collected data is immediately transmitted to a server. The server analyzes this data using artificial intelligence technology and analytical tools (e.g., TENSORFLOW® and OpenCV) to detect anomalies. If an anomaly is detected, the server generates the optimal maintenance procedure based on the analysis results and presents it to the operator.

[0105] The terminals include smartphones and head-mounted displays. Maintenance procedures transmitted from the server are visually displayed on the terminal using augmented reality technology. This is achieved using Unity's ARFoundation, among other technologies. Workers can use this information to perform repair work quickly and safely. Furthermore, workers are immediately notified if any abnormalities are detected.

[0106] As a concrete example, consider the inspection of solar panels installed on the roof of a large warehouse. An unmanned aerial vehicle (UAV) patrols the area, identifying dirt and damaged parts of the panels. A server analyzes this information and sends a notification if an anomaly is detected. Workers can safely perform the task using a head-mounted display, following repair instructions displayed in augmented reality.

[0107] An example of a prompt message is: "Please describe a high-altitude inspection system using unmanned aerial vehicles. Consider its applications comprehensively and describe in detail how it contributes to safety and efficiency, especially its application to security services." This invention will enable even unskilled personnel to quickly perform maintenance on communication infrastructure, significantly reducing the burden on technicians.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The server receives weather data and terrain information as input. It analyzes this data to design a flight path for safe and efficient flight and generates a flight plan. The generated flight plan is then transmitted to the unmanned aerial vehicle (UAV).

[0111] Step 2:

[0112] The unmanned aerial vehicle receives the flight plan and begins autonomous flight. It collects ambient environmental data and video data from communication devices using cameras and sensors, and transmits this data to a server in real time.

[0113] Step 3:

[0114] The server takes received video and environmental data as input and performs data analysis using artificial intelligence technology. The AI ​​model responsible for anomaly detection uses, for example, TensorFlow or OpenCV to determine whether or not an anomaly is present. Based on the analysis results, the server generates the optimal maintenance procedure if an anomaly is found. This result is sent to the worker's terminal.

[0115] Step 4:

[0116] The terminal visually displays the analysis results and maintenance procedures received from the server using augmented reality technology. Workers can then review the maintenance procedures as an AR overlay via a head-mounted display or similar device and perform repair work. Using Unity's ARFoundation makes the operating procedures easier to understand visually.

[0117] Step 5:

[0118] Based on notification information displayed on the device, users can quickly identify the repair location and perform maintenance work. By referring to augmented reality visual information, even inexperienced users can perform maintenance work efficiently.

[0119] In this way, by following all steps, the unmanned aerial vehicle, the data analysis server, and the worker's terminal work in coordination, making maintenance work on communication equipment more efficient.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention relates to a system comprising an unmanned aerial vehicle, a server for data analysis, a terminal for use by an operator, and an emotion engine for recognizing user emotions. The roles and specific operations of each component are described below.

[0122] The server uses location information and environmental data from the communication device to formulate a flight plan for the unmanned aerial vehicle (UAV). This plan specifies a safe and efficient route while adapting to environmental conditions. The UAV performs autonomous flight according to the plan received from the server, collecting video data using sensors and cameras during patrols. This data is sent back to the server, where artificial intelligence is used to analyze for any anomalies.

[0123] The terminal provides the worker with optimal maintenance procedures generated based on the analyzed data. These procedures are also presented visually using augmented reality technology, allowing the worker to follow them more intuitively.

[0124] A distinctive feature of this invention is the incorporation of an emotion engine. The emotion engine is built into the terminal and analyzes the worker's voice, facial expressions, and behavioral patterns to recognize their current emotional state. The server receives this emotional data and adjusts the method of providing maintenance procedures and the content of support according to the worker's mental state.

[0125] For example, if the emotion engine detects that a worker is stressed, the terminal will display a more detailed procedural guide, and the server will add response time and confirmation alerts to assist the worker. In this way, creating an environment where workers can work with peace of mind improves the safety and efficiency of maintenance work.

[0126] This system reduces the risks faced by technicians and provides comprehensive support for fast and accurate maintenance work.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The server creates a flight plan for the unmanned aerial vehicle (UAV) based on the location of the communication device and current environmental data. During this process, it considers weather conditions and terrain information to select a safe and efficient route. The generated flight plan is then transmitted to the UAV.

[0130] Step 2:

[0131] The unmanned aerial vehicle (UAV) begins flying autonomously according to the flight plan received from the server. During flight, it collects video data from its communication devices and other sensor data in real time and transmits this data to the server.

[0132] Step 3:

[0133] The server uses the received data to perform analysis using artificial intelligence. The AI ​​analyzes the image data and detects anomalies. For example, it identifies loose or damaged parts of cables and reports the anomaly.

[0134] Step 4:

[0135] The server generates the optimal maintenance procedure based on the analysis results. This procedure includes specific repair methods and the tools to be used. This information is then sent to the worker's terminal.

[0136] Step 5:

[0137] The user, acting as the worker, begins the task based on the information provided by the terminal. The terminal assists the worker by visually displaying the repair location and procedure using augmented reality technology.

[0138] Step 6:

[0139] The emotion engine built into the terminal analyzes the worker's voice and facial expressions to evaluate their emotional state in real time. For example, if anxiety or lack of concentration is detected, the terminal increases the amount of work support information.

[0140] Step 7:

[0141] The server receives feedback from the emotion engine and adjusts the way work procedures are presented and the level of detail in maintenance instructions. It provides appropriate support so that users can work with confidence.

[0142] Step 8:

[0143] The user completes the task with assistance from the emotion engine and the server. After completion, they submit a report from their terminal, and the server records the results in a database. This data will be used for future analysis and to improve the accuracy of the assistance.

[0144] (Example 2)

[0145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] In the inspection of communication equipment at high altitudes using unmanned aerial vehicles, there is a need for anomaly detection and efficient maintenance work. However, there is a lack of support that takes into account the emotional state of the workers, and the safety and efficiency of maintenance work are not adequately ensured.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes means for collecting information using an unmanned aerial vehicle, means for analyzing the collected information with artificial intelligence and detecting anomalies, means for suggesting optimal maintenance work based on the analysis results, and means for recognizing the emotional state of the worker and adjusting the maintenance work based on that data. This enables improvements in the safety and efficiency of maintenance work, as well as mental support for the worker.

[0149] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and collects data using onboard sensors and cameras.

[0150] "Communication equipment" refers to devices installed at high altitudes for communication purposes, and is hardware used for sending and receiving data.

[0151] "Information" refers to data used for analysis, such as image data and environmental data collected by unmanned aerial vehicles.

[0152] "Artificial intelligence" is a technology that mimics the intelligent functions of computer systems, and in particular, it refers to programs that have the ability to detect anomalies based on data.

[0153] "Maintenance work" refers to tasks performed to maintain the functionality of communication equipment and to repair it, and is an action taken to efficiently resolve problems.

[0154] An "emotion engine" is a system that analyzes the worker's voice, facial expressions, etc., to recognize their emotional state at that time.

[0155] "Means" is a term that refers to a device, a group of devices, or a method used to achieve a specific function or purpose.

[0156] Augmented reality technology is a technique that overlays digital information onto the real world and is used to provide visual assistance to workers.

[0157] Modes for carrying out the invention

[0158] The present invention is a system that includes an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine for recognizing the user's emotions.

[0159] System Configuration

[0160] Server operation

[0161] The server receives and stores data transmitted from the unmanned aerial vehicle (UAV) and performs analysis using artificial intelligence. The software used includes generative AI models, which are employed for pattern recognition and anomaly detection in the data. It also accesses external APIs to obtain weather conditions and geographical data, and creates flight plans for the UAV. This plan identifies the optimal flight path and instructs the UAV accordingly.

[0162] Operation of unmanned aerial vehicles

[0163] The unmanned aerial vehicle (UAV) flies autonomously and collects data from communication equipment installed at high altitudes. The hardware used for collection includes high-resolution cameras and various sensors. Flight plans are provided by a server, and the UAV monitors the environment according to this plan and transmits the necessary information to the server.

[0164] Terminal operation

[0165] The terminal presents the worker with the most suitable maintenance tasks based on analysis results received from the server. Augmented reality technology is used for the presentation, enhancing visual assistance and enabling intuitive work procedures. In addition, a built-in emotion engine analyzes the worker's voice and facial expressions to assess their mental state. If the worker is experiencing stress, support is enhanced by providing more detailed explanations and appropriate alerts.

[0166] Specific example

[0167] For example, while a worker is inspecting an unmanned aerial vehicle, the server can detect anomalies in real time based on a generated AI model. The worker's terminal displays details of the anomaly and specific countermeasures via augmented reality. An emotion engine, sensing the worker's tension from their tone of voice, displays tips on the terminal to help them relax.

[0168] Example of a prompt

[0169] "Please explain how maintenance procedures can be dynamically adjusted based on the emotional state of the workers."

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The server retrieves the latest environmental data from external weather APIs and geographic information databases. Inputs include location information and the types of environmental data required, and queries are executed against the databases and APIs. Outputs include environmental data necessary for developing safe flight plans for unmanned aerial vehicles (UAVs). This data is used to execute algorithms that determine the optimal flight route for the UAV. Specific operations include weather pattern analysis and terrain analysis.

[0173] Step 2:

[0174] The unmanned aerial vehicle (UAV) begins autonomous flight based on a flight plan received from a server. Inputs include the flight plan transmitted from the server and real-time environmental data collected by onboard sensors and cameras. During flight, the UAV periodically takes photos and videos and collects other environmental data with its sensors. Outputs include sending this data as data packets to the server. Specific operations include high-resolution video capture and temperature and humidity measurement.

[0175] Step 3:

[0176] The server receives data transmitted from the unmanned aerial vehicle (UAV) and performs analysis for anomaly detection using a generative AI model. Inputs include image data and environmental data collected by the UAV. The server inputs this data into an AI algorithm and compares it to known anomaly patterns. The output provides the presence or absence of anomalies and their details as analysis results. Specific operations include pattern matching utilizing image recognition technology.

[0177] Step 4:

[0178] The terminal receives analysis results from the server and generates maintenance procedures to present to the operator. Input includes a database of anomaly detection results and repair procedures provided by the server. The terminal uses augmented reality technology to visually present the procedures in a way that the operator can intuitively understand. Output provides the operator with visually guided maintenance procedures. Specific actions include a visual overlay using AR markers.

[0179] Step 5:

[0180] The terminal uses an emotion engine to analyze the worker's voice and facial expressions and assess their emotional state. Inputs include real-time audio data and video feeds. Based on the analysis results, the server adjusts work procedures and alerts as needed, and provides customized instructions to the worker as output. Specific actions include the dynamic generation of alerts and support messages to address different emotional states.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] In patrol systems using unmanned aerial vehicles, there is a need for means to accurately detect abnormalities in communication equipment and to ensure that workers can perform maintenance work with peace of mind. Furthermore, it is necessary to flexibly adjust maintenance procedures according to the emotional state of the workers to guarantee an efficient and safe working environment.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to the worker based on the analysis results; and means equipped with an emotion engine that recognizes the worker's emotional state and adjusts the presented maintenance procedures according to the worker's emotions. This enables anomaly detection of communication equipment, improved efficiency of maintenance work, and enhanced safety.

[0186] An "unmanned aerial vehicle" is an autonomous flying machine used to patrol communication equipment installed at high altitudes and efficiently collect data.

[0187] "Artificial intelligence that analyzes data and detects anomalies" is a computer program that uses collected data to identify irregular patterns and anomalies.

[0188] "A means of generating and presenting optimal maintenance procedures to workers" refers to a method of formulating the most efficient repair and maintenance methods based on analysis results and providing them to workers.

[0189] The "emotion engine" is an analytical device that grasps the psychological state of a worker from their facial expressions, voice, and movement data, and provides appropriate support according to that state.

[0190] Augmented reality technology is a technique that overlays digital data onto visual information from the real world, allowing workers to intuitively understand the information.

[0191] The system for carrying out this invention consists of an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine that recognizes the operator's emotions.

[0192] The server first develops a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This plan is refined using environmental data to provide a safe and efficient route. The UAV flies autonomously, collecting data using cameras and sensors during the patrol. This data is transmitted to the server, where artificial intelligence analyzes the data and detects any anomalies.

[0193] Based on the analysis results, the server generates the optimal maintenance procedure. This procedure is displayed on the worker's terminal and presented visually using augmented reality technology, making it easier for the worker to intuitively understand the procedure. The emotion engine built into the terminal recognizes the worker's emotional state based on voice, facial expressions, and behavioral data, and the server uses this emotional data to provide the maintenance procedure in a way that is appropriate for the worker. For example, if the worker is nervous, detailed guides and alerts are added to support the work.

[0194] Implementing this system will enable the detection of abnormalities in communication devices, provide workers with flexible maintenance procedures, and improve worker safety and work efficiency.

[0195] A concrete example is maintenance work during the installation of new equipment in a factory. In this scenario, an unmanned aerial vehicle (UAV) inspects the equipment and collects data. If a server detects an anomaly, it provides detailed instructions to the worker via a terminal. If the worker is stressed, an emotion engine detects this and provides additional support.

[0196] An example of a prompt for a generative AI model is, "Develop effective support strategies based on sentiment data as workers perform maintenance on new equipment."

[0197] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0198] Step 1:

[0199] The server develops flight plans for unmanned aerial vehicles (UAVs) to patrol communication equipment installed at high altitudes. Inputs include environmental data and historical flight data, and the server calculates and outputs the optimal route based on this data. Specifically, it automatically generates safe and efficient routes, taking into account environmental conditions such as wind speed and temperature.

[0200] Step 2:

[0201] The unmanned aerial vehicle (UAV) performs autonomous flight according to a flight plan sent from a server. Its input is flight plan data from the server; it uses this as a basis for actual flight, collecting images and sensor data. The collected data is then transferred back to the server as output. Specific operations include obstacle avoidance and real-time position adjustment during flight.

[0202] Step 3:

[0203] The server analyzes data sent from unmanned aerial vehicles. Inputs include image data and sensor data, which are analyzed using an artificial intelligence model to detect anomalies. Outputs are reports indicating the presence or absence of anomalies. For example, image processing techniques are used to check the surface condition of equipment.

[0204] Step 4:

[0205] The server generates the optimal maintenance procedure based on the analysis results and sends it to the terminal. The input is the anomaly detection result, and based on this, it formulates and outputs an appropriate maintenance procedure. Specifically, it extracts the necessary tasks from the equipment manual and generates a procedure document.

[0206] Step 5:

[0207] The terminal presents the received maintenance procedures to the worker using augmented reality technology. The input is maintenance procedure data from the server, which is then displayed and output as visual information. Specifically, guidelines are displayed in the worker's field of view via an AR headset.

[0208] Step 6:

[0209] An emotion engine built into the terminal recognizes the worker's emotional state. Inputs include the worker's voice, facial expressions, and behavioral data, which are analyzed to identify and output the emotional state. Specifically, it uses voice analysis and facial recognition technology to detect stress and anxiety.

[0210] Step 7:

[0211] The server analyzes emotional state data obtained from the emotion engine and adjusts how maintenance procedures are presented. The input is the recognized emotional data, and based on this, it adjusts and outputs support procedures tailored to the worker. Specifically, it adjusts the level of detail in explanations and adds alerts.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] The system of the present invention includes an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by operators. The roles and operations of each component will be described in detail below.

[0229] The server creates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This flight plan includes route information designed to allow the UAV to patrol the destination safely and efficiently, and also takes weather and terrain data into consideration. Based on this information, the server sends real-time instructions to the UAV.

[0230] The unmanned aerial vehicle (UAV) flies autonomously based on instructions from the server, patrolling the area around the communication device. During this time, it collects video and environmental data from the communication device using its equipped cameras and sensors. This collected data is transmitted to the server in real time.

[0231] The terminal is a device held by the worker and functions as an interface that displays analysis results and maintenance procedures from the server. The server analyzes the transmitted data using artificial intelligence to detect anomalies and identify their causes. The analysis results, along with the optimal maintenance procedures, are sent to the terminal, and the worker uses this information to repair or adjust the communication equipment.

[0232] As a concrete example, if a malfunction occurs in a communication device installed at a high altitude, the unmanned aerial vehicle (UAV) will patrol the area and send detailed video footage back to the server. Based on the video, the server will identify the abnormality, such as a damaged cable or a loose connection, and create specific procedures to correct it. Workers can then safely and efficiently perform repairs by checking the graphical procedures displayed using AR technology through a terminal.

[0233] This system enables the maintenance of communication equipment quickly and accurately, regardless of the operator's experience. The introduction of this system will enhance the reliability of the communication infrastructure and reduce the workload on technicians.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] Based on the location information of communication devices requiring patrol, the server generates a flight plan for the unmanned aerial vehicle (UAV), taking into account weather data and current operational conditions. This flight plan includes details of an efficient and safe patrol route.

[0237] Step 2:

[0238] The unmanned aerial vehicle (UAV) begins flight autonomously according to the flight plan received from the server via the terminal. During flight, it collects video data of the surroundings of the communication device using high-precision cameras and various sensors.

[0239] Step 3:

[0240] The server receives data transmitted in real time from the unmanned aerial vehicle. The server's artificial intelligence analysis engine processes this data in real time and uses image analysis algorithms to detect anomalies in the communication equipment.

[0241] Step 4:

[0242] Based on the detected anomaly, the server generates the optimal repair procedure by comparing it with past repair history. This repair procedure clearly specifies the particular work methods and tools to be used.

[0243] Step 5:

[0244] Repair instructions generated from the server are sent to the user's (worker's) terminal. The terminal displays the maintenance instructions and necessary information visually using augmented reality technology.

[0245] Step 6:

[0246] The user, acting as the operator, utilizes the information and AR guidance provided on the terminal to efficiently and safely repair and adjust communication equipment. This allows the operator to quickly resolve any problems.

[0247] (Example 1)

[0248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] Communication equipment installed at high altitudes is difficult to inspect and maintain, requiring early detection and rapid response to any abnormalities. However, conventional methods are dependent on the skill level of the workers and weather conditions, making efficient maintenance challenging. Furthermore, there is a need for automated abnormality detection and the provision of efficient maintenance procedures.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0251] In this invention, the server includes means for patrolling communication equipment using a data processing device and collecting information, evaluation means for analyzing the collected information using a machine learning model and detecting anomalies, means for generating and providing optimal maintenance actions to the operator based on the analysis results, and means for creating a flight plan using a machine learning model. This enables early detection of anomalies and the provision of efficient maintenance procedures, independent of the individual capabilities of the operator.

[0252] A "data processing device" is an autonomous, mobile device designed to patrol communication equipment installed at high altitudes and collect information about the surrounding environment.

[0253] "Information" refers to data about the surrounding environment and state of communication equipment, including images and audio acquired through cameras and sensors, as well as environmental measurements.

[0254] A "machine learning model" is a set of algorithms used to analyze large amounts of data, identify patterns, and detect anomalies, and includes artificial intelligence technology.

[0255] "Evaluation means" refers to a system process that uses a machine learning model based on collected information to determine the normality of communication equipment and identify abnormalities.

[0256] "Maintenance actions" refer to an action plan that outlines the work and processes necessary to correct malfunctions in communication equipment, and includes specific instructions for workers.

[0257] A "flight plan" is a plan that outlines the route information a data processing unit will take when patrolling communication equipment, and is designed to maximize safety and efficiency.

[0258] "Means to be provided to the worker" refers to technologies for appropriately presenting the analyzed results and maintenance actions to the worker, and includes visual display methods.

[0259] The embodiments for carrying out the present invention are described below.

[0260] This system includes a data processing unit (unmanned aerial vehicle), a computer system (server) for controlling it and analyzing data, and an operating device (terminal) used by operators. This allows for the rapid detection of abnormalities in communication equipment and enables appropriate maintenance.

[0261] The server generates a flight plan for the data processing unit to patrol the communication equipment. Specifically, the server uses a generation AI model to analyze real-time weather data and terrain information to design a safe and efficient flight path. Furthermore, the server sends commands to the data processing unit via a communication protocol. The MQTT protocol is used as needed.

[0262] The data processing unit autonomously patrols the area around the communication equipment according to instructions from the server, and returns information collected by its equipped cameras and sensors to the server in real time. This information includes image data, environmental data, and acoustic data.

[0263] The server uses the received information to detect anomalies with a machine learning model and generates maintenance actions based on the results. The evaluation method utilizes algorithms suitable for image analysis, such as convolutional neural networks (CNNs).

[0264] Subsequently, the server transmits the analyzed anomaly information and maintenance procedures to the terminal, which then visually presents this information to the worker. Specifically, augmented reality technology is used to display instructions in a format that is easy for the worker to understand.

[0265] For example, if an anomaly is detected in communication equipment installed at a high altitude, a data processing unit flies over the area and collects detailed video footage using a camera. The server analyzes this data, and if an anomaly is identified, it provides the worker with the most suitable repair procedure via a terminal.

[0266] An example of a prompt message for a generative AI model is, "Prepare a means to patrol communication devices and detect anomalies from the collected information."

[0267] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0268] Step 1:

[0269] The server generates flight plans using a generative AI model. It receives real-time weather data and terrain information as input. This data is analyzed to calculate a safe and efficient route. As output, it creates a flight plan including route information and sends it to a data processing unit. Specifically, the server obtains wind speed and rainfall information via a weather API and performs route optimization based on this information.

[0270] Step 2:

[0271] The data processing unit patrols the area around the communication equipment according to the flight plan received from the server. It receives command routes from the server as input and transmits video and environmental data from the communication equipment to the server as output. Specific operations include acquiring detailed images and measurement data in real time during patrols using cameras and sensors.

[0272] Step 3:

[0273] The server analyzes video and environmental data received from the data processing unit. It receives data from multiple sensors as input and uses a machine learning model to detect anomalies. As output, it generates anomaly identification and diagnostic results, and plans maintenance actions based on these results. Specifically, the server utilizes a convolutional neural network (CNN) to identify anomalies in the video data.

[0274] Step 4:

[0275] The server transmits analysis results and maintenance actions to the terminal. Using the diagnostic results from the analysis as input, it generates specific instructions for the worker as output. These specific actions include creating documents containing repair procedures and important points to note.

[0276] Step 5:

[0277] The terminal provides information to the worker based on instructions from the server. It receives maintenance instructions from the server as input and displays the information in a visually understandable format as output. Specifically, it utilizes augmented reality (AR) technology to display repair locations as 3D models, thereby supporting the worker's understanding.

[0278] Step 6:

[0279] The user (worker) performs the actual maintenance work based on the information displayed on the terminal. The input is the instruction information from the terminal, and the output is the restoration of the communication equipment to normal operation. Specific actions include following the steps indicated by the terminal in order and reporting the completion of the work to the server upon completion.

[0280] (Application Example 1)

[0281] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0282] The inspection and maintenance work of conventional communication devices involved working at heights, which posed safety challenges and imposed a heavy burden on workers. In addition, since the detection of abnormalities and the determination of repair methods relied on the experience of workers, there was a lack of efficiency and consistency. As a result, it was difficult to respond quickly, and there was a risk of reducing the reliability of the communication infrastructure.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0284] In this invention, the server includes means for inspecting a communication device installed at a height using an unmanned aerial vehicle and collecting data, analytical means using artificial intelligence for analyzing the data and detecting abnormalities, means for generating an optimal maintenance procedure based on the analysis results and presenting it to the worker, and means for immediately notifying when an abnormality is detected and visually displaying repair and confirmation procedures using augmented reality technology. As a result, the inspection and maintenance of communication devices can be performed safely and efficiently, reducing the burden on workers and improving the reliability of the communication infrastructure.

[0285] An "unmanned aerial vehicle" is a device that flies remotely or autonomously and performs certain functions.

[0286] A "communication device" is an electronic device installed at a height that transmits and receives data.

[0287] "Data" refers to information on the communication device and its surrounding conditions collected by the unmanned aerial vehicle.

[0288] "Artificial intelligence" is a technology in which a computer program imitates human intelligence and analyzes data to detect abnormalities.

[0289] "Analytical means" is a method or mechanism for evaluating data and determining the presence or absence of abnormalities.

[0290] A "maintenance procedure" is a standard operating procedure set to correct abnormalities in a communication device.

[0291] Augmented reality technology is a technology that overlays virtual information onto the real world's field of view.

[0292] A "smartphone" is a portable information processing device that allows for communication and the use of various applications.

[0293] A "head-mounted display" is a device that displays images in the wearer's field of vision.

[0294] "Notification" refers to the act of communicating information to inform workers of the occurrence of an abnormality.

[0295] The present invention comprises an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by an operator. The following method is used to realize this system.

[0296] The server generates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. The server considers weather data and terrain information to create route information that enables autonomous flight, and sends real-time instructions to the UAV. In this process, GIS data is used to design a safe and efficient route.

[0297] The unmanned aerial vehicle (UAV) uses its onboard cameras and sensors to collect video data from communication devices and data about the surrounding environment. The collected data is immediately transmitted to a server. The server analyzes this data using artificial intelligence technology and analytical tools (e.g., TensorFlow and OpenCV) to detect anomalies. If an anomaly is detected, the server generates the optimal maintenance procedure based on the analysis results and presents it to the operator.

[0298] The terminals include smartphones and head-mounted displays. Maintenance procedures transmitted from the server are visually displayed on the terminal using augmented reality technology. This is achieved using Unity's ARFoundation, among other technologies. Workers can use this information to perform repair work quickly and safely. Furthermore, workers are immediately notified if any abnormalities are detected.

[0299] As a concrete example, consider the inspection of solar panels installed on the roof of a large warehouse. An unmanned aerial vehicle (UAV) patrols the area, identifying dirt and damaged parts of the panels. A server analyzes this information and sends a notification if an anomaly is detected. Workers can safely perform the task using a head-mounted display, following repair instructions displayed in augmented reality.

[0300] An example of a prompt message is: "Please describe a high-altitude inspection system using unmanned aerial vehicles. Consider its applications comprehensively and describe in detail how it contributes to safety and efficiency, especially its application to security services." This invention will enable even unskilled personnel to quickly perform maintenance on communication infrastructure, significantly reducing the burden on technicians.

[0301] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0302] Step 1:

[0303] The server receives weather data and terrain information as input. It analyzes this data to design a flight path for safe and efficient flight and generates a flight plan. The generated flight plan is then transmitted to the unmanned aerial vehicle (UAV).

[0304] Step 2:

[0305] The unmanned aerial vehicle receives the flight plan and begins autonomous flight. It collects ambient environmental data and video data from communication devices using cameras and sensors, and transmits this data to a server in real time.

[0306] Step 3:

[0307] The server uses the received video data and environmental data as input and performs data analysis using artificial intelligence technology. The AI model responsible for anomaly detection uses, for example, TensorFlow or OpenCV to determine the presence or absence of anomalies. Based on the analysis results, the server generates an optimal maintenance procedure if there is an anomaly. This result is sent to the operator's terminal.

[0308] Step 4:

[0309] On the terminal, the analysis results and maintenance procedures received from the server are visually displayed using augmented reality technology. The operator can confirm the maintenance procedures as an AR overlay through a head-mounted display or the like and perform repair work. By using Unity's ARFoundation, the operation procedures become easier to understand visually.

[0310] Step 5:

[0311] The user quickly identifies the repair location based on the notification information displayed on the terminal and performs maintenance work. By referring to the visual information by augmented reality, the user can perform maintenance work efficiently even if they are unskilled.

[0312] In this way, by processing all the steps, the unmanned aerial vehicle, the data analysis server, and the operator's terminal cooperate to improve the efficiency of the maintenance work of the communication device.

[0313] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0314] This invention relates to a system comprising an unmanned aerial vehicle, a server for data analysis, a terminal for use by an operator, and an emotion engine for recognizing user emotions. The roles and specific operations of each component are described below.

[0315] The server uses location information and environmental data from the communication device to formulate a flight plan for the unmanned aerial vehicle (UAV). This plan specifies a safe and efficient route while adapting to environmental conditions. The UAV performs autonomous flight according to the plan received from the server, collecting video data using sensors and cameras during patrols. This data is sent back to the server, where artificial intelligence is used to analyze for any anomalies.

[0316] The terminal provides the worker with optimal maintenance procedures generated based on the analyzed data. These procedures are also presented visually using augmented reality technology, allowing the worker to follow them more intuitively.

[0317] A distinctive feature of this invention is the incorporation of an emotion engine. The emotion engine is built into the terminal and analyzes the worker's voice, facial expressions, and behavioral patterns to recognize their current emotional state. The server receives this emotional data and adjusts the method of providing maintenance procedures and the content of support according to the worker's mental state.

[0318] For example, if the emotion engine detects that a worker is stressed, the terminal will display a more detailed procedural guide, and the server will add response time and confirmation alerts to assist the worker. In this way, creating an environment where workers can work with peace of mind improves the safety and efficiency of maintenance work.

[0319] This system reduces the risks faced by technicians and provides comprehensive support for fast and accurate maintenance work.

[0320] The following describes the processing flow.

[0321] Step 1:

[0322] The server creates a flight plan for the unmanned aerial vehicle (UAV) based on the location of the communication device and current environmental data. During this process, it considers weather conditions and terrain information to select a safe and efficient route. The generated flight plan is then transmitted to the UAV.

[0323] Step 2:

[0324] The unmanned aerial vehicle (UAV) begins flying autonomously according to the flight plan received from the server. During flight, it collects video data from its communication devices and other sensor data in real time and transmits this data to the server.

[0325] Step 3:

[0326] The server uses the received data to perform analysis using artificial intelligence. The AI ​​analyzes the image data and detects anomalies. For example, it identifies loose or damaged parts of cables and reports the anomaly.

[0327] Step 4:

[0328] The server generates the optimal maintenance procedure based on the analysis results. This procedure includes specific repair methods and the tools to be used. This information is then sent to the worker's terminal.

[0329] Step 5:

[0330] The user, acting as the worker, begins the task based on the information provided by the terminal. The terminal assists the worker by visually displaying the repair location and procedure using augmented reality technology.

[0331] Step 6:

[0332] The emotion engine built into the terminal analyzes the worker's voice and facial expressions to evaluate their emotional state in real time. For example, if anxiety or lack of concentration is detected, the terminal increases the amount of work support information.

[0333] Step 7:

[0334] The server receives feedback from the emotion engine and adjusts the way work procedures are presented and the level of detail in maintenance instructions. It provides appropriate support so that users can work with confidence.

[0335] Step 8:

[0336] The user completes the task with assistance from the emotion engine and the server. After completion, they submit a report from their terminal, and the server records the results in a database. This data will be used for future analysis and to improve the accuracy of the assistance.

[0337] (Example 2)

[0338] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0339] In the inspection of communication equipment at high altitudes using unmanned aerial vehicles, there is a need for anomaly detection and efficient maintenance work. However, there is a lack of support that takes into account the emotional state of the workers, and the safety and efficiency of maintenance work are not adequately ensured.

[0340] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0341] In this invention, the server includes means for collecting information using an unmanned aerial vehicle, means for analyzing the collected information with artificial intelligence and detecting anomalies, means for suggesting optimal maintenance work based on the analysis results, and means for recognizing the emotional state of the worker and adjusting the maintenance work based on that data. This enables improvements in the safety and efficiency of maintenance work, as well as mental support for the worker.

[0342] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and collects data using onboard sensors and cameras.

[0343] "Communication equipment" refers to devices installed at high altitudes for communication purposes, and is hardware used for sending and receiving data.

[0344] "Information" refers to data used for analysis, such as image data and environmental data collected by unmanned aerial vehicles.

[0345] "Artificial intelligence" is a technology that mimics the intelligent functions of computer systems, and in particular, it refers to programs that have the ability to detect anomalies based on data.

[0346] "Maintenance work" refers to tasks performed to maintain the functionality of communication equipment and to repair it, and is an action taken to efficiently resolve problems.

[0347] An "emotion engine" is a system that analyzes the worker's voice, facial expressions, etc., to recognize their emotional state at that time.

[0348] "Means" is a term that refers to a device, a group of devices, or a method used to achieve a specific function or purpose.

[0349] Augmented reality technology is a technique that overlays digital information onto the real world and is used to provide visual assistance to workers.

[0350] Modes for carrying out the invention

[0351] The present invention is a system that includes an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine for recognizing the user's emotions.

[0352] System Configuration

[0353] Server operation

[0354] The server receives and stores data transmitted from the unmanned aerial vehicle (UAV) and performs analysis using artificial intelligence. The software used includes generative AI models, which are employed for pattern recognition and anomaly detection in the data. It also accesses external APIs to obtain weather conditions and geographical data, and creates flight plans for the UAV. This plan identifies the optimal flight path and instructs the UAV accordingly.

[0355] Operation of unmanned aerial vehicles

[0356] The unmanned aerial vehicle (UAV) flies autonomously and collects data from communication equipment installed at high altitudes. The hardware used for collection includes high-resolution cameras and various sensors. Flight plans are provided by a server, and the UAV monitors the environment according to this plan and transmits the necessary information to the server.

[0357] Terminal operation

[0358] The terminal presents the worker with the most suitable maintenance tasks based on analysis results received from the server. Augmented reality technology is used for the presentation, enhancing visual assistance and enabling intuitive work procedures. In addition, a built-in emotion engine analyzes the worker's voice and facial expressions to assess their mental state. If the worker is experiencing stress, support is enhanced by providing more detailed explanations and appropriate alerts.

[0359] Specific example

[0360] For example, while a worker is inspecting an unmanned aerial vehicle, the server can detect anomalies in real time based on a generated AI model. The worker's terminal displays details of the anomaly and specific countermeasures via augmented reality. An emotion engine, sensing the worker's tension from their tone of voice, displays tips on the terminal to help them relax.

[0361] Example of a prompt

[0362] "Please explain how maintenance procedures can be dynamically adjusted based on the emotional state of the workers."

[0363] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0364] Step 1:

[0365] The server retrieves the latest environmental data from external weather APIs and geographic information databases. Inputs include location information and the types of environmental data required, and queries are executed against the databases and APIs. Outputs include environmental data necessary for developing safe flight plans for unmanned aerial vehicles (UAVs). This data is used to execute algorithms that determine the optimal flight route for the UAV. Specific operations include weather pattern analysis and terrain analysis.

[0366] Step 2:

[0367] The unmanned aerial vehicle (UAV) begins autonomous flight based on a flight plan received from a server. Inputs include the flight plan transmitted from the server and real-time environmental data collected by onboard sensors and cameras. During flight, the UAV periodically takes photos and videos and collects other environmental data with its sensors. Outputs include sending this data as data packets to the server. Specific operations include high-resolution video capture and temperature and humidity measurement.

[0368] Step 3:

[0369] The server receives data transmitted from the unmanned aerial vehicle (UAV) and performs analysis for anomaly detection using a generative AI model. Inputs include image data and environmental data collected by the UAV. The server inputs this data into an AI algorithm and compares it to known anomaly patterns. The output provides the presence or absence of anomalies and their details as analysis results. Specific operations include pattern matching utilizing image recognition technology.

[0370] Step 4:

[0371] The terminal receives analysis results from the server and generates maintenance procedures to present to the operator. Input includes a database of anomaly detection results and repair procedures provided by the server. The terminal uses augmented reality technology to visually present the procedures in a way that the operator can intuitively understand. Output provides the operator with visually guided maintenance procedures. Specific actions include a visual overlay using AR markers.

[0372] Step 5:

[0373] The terminal uses an emotion engine to analyze the worker's voice and facial expressions and assess their emotional state. Inputs include real-time audio data and video feeds. Based on the analysis results, the server adjusts work procedures and alerts as needed, and provides customized instructions to the worker as output. Specific actions include the dynamic generation of alerts and support messages to address different emotional states.

[0374] (Application Example 2)

[0375] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0376] In patrol systems using unmanned aerial vehicles, there is a need for means to accurately detect abnormalities in communication equipment and to ensure that workers can perform maintenance work with peace of mind. Furthermore, it is necessary to flexibly adjust maintenance procedures according to the emotional state of the workers to guarantee an efficient and safe working environment.

[0377] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0378] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to the worker based on the analysis results; and means equipped with an emotion engine that recognizes the worker's emotional state and adjusts the presented maintenance procedures according to the worker's emotions. This enables anomaly detection of communication equipment, improved efficiency of maintenance work, and enhanced safety.

[0379] An "unmanned aerial vehicle" is an autonomous flying machine used to patrol communication equipment installed at high altitudes and efficiently collect data.

[0380] "Artificial intelligence that analyzes data and detects anomalies" is a computer program that uses collected data to identify irregular patterns and anomalies.

[0381] "A means of generating and presenting optimal maintenance procedures to workers" refers to a method of formulating the most efficient repair and maintenance methods based on analysis results and providing them to workers.

[0382] The "emotion engine" is an analytical device that grasps the psychological state of a worker from their facial expressions, voice, and movement data, and provides appropriate support according to that state.

[0383] Augmented reality technology is a technique that overlays digital data onto visual information from the real world, allowing workers to intuitively understand the information.

[0384] The system for carrying out this invention consists of an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine that recognizes the operator's emotions.

[0385] The server first develops a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This plan is refined using environmental data to provide a safe and efficient route. The UAV flies autonomously, collecting data using cameras and sensors during the patrol. This data is transmitted to the server, where artificial intelligence analyzes the data and detects any anomalies.

[0386] Based on the analysis results, the server generates the optimal maintenance procedure. This procedure is displayed on the worker's terminal and presented visually using augmented reality technology, making it easier for the worker to intuitively understand the procedure. The emotion engine built into the terminal recognizes the worker's emotional state based on voice, facial expressions, and behavioral data, and the server uses this emotional data to provide the maintenance procedure in a way that is appropriate for the worker. For example, if the worker is nervous, detailed guides and alerts are added to support the work.

[0387] Implementing this system will enable the detection of abnormalities in communication devices, provide workers with flexible maintenance procedures, and improve worker safety and work efficiency.

[0388] A concrete example is maintenance work during the installation of new equipment in a factory. In this scenario, an unmanned aerial vehicle (UAV) inspects the equipment and collects data. If a server detects an anomaly, it provides detailed instructions to the worker via a terminal. If the worker is stressed, an emotion engine detects this and provides additional support.

[0389] An example of a prompt for a generative AI model is, "Develop effective support strategies based on sentiment data as workers perform maintenance on new equipment."

[0390] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0391] Step 1:

[0392] The server develops flight plans for unmanned aerial vehicles (UAVs) to patrol communication equipment installed at high altitudes. Inputs include environmental data and historical flight data, and the server calculates and outputs the optimal route based on this data. Specifically, it automatically generates safe and efficient routes, taking into account environmental conditions such as wind speed and temperature.

[0393] Step 2:

[0394] The unmanned aerial vehicle (UAV) performs autonomous flight according to a flight plan sent from a server. Its input is flight plan data from the server; it uses this as a basis for actual flight, collecting images and sensor data. The collected data is then transferred back to the server as output. Specific operations include obstacle avoidance and real-time position adjustment during flight.

[0395] Step 3:

[0396] The server analyzes data sent from unmanned aerial vehicles. Inputs include image data and sensor data, which are analyzed using an artificial intelligence model to detect anomalies. Outputs are reports indicating the presence or absence of anomalies. For example, image processing techniques are used to check the surface condition of equipment.

[0397] Step 4:

[0398] The server generates the optimal maintenance procedure based on the analysis results and sends it to the terminal. The input is the anomaly detection result, and based on this, it formulates and outputs an appropriate maintenance procedure. Specifically, it extracts the necessary tasks from the equipment manual and generates a procedure document.

[0399] Step 5:

[0400] The terminal presents the received maintenance procedures to the worker using augmented reality technology. The input is maintenance procedure data from the server, which is then displayed and output as visual information. Specifically, guidelines are displayed in the worker's field of view via an AR headset.

[0401] Step 6:

[0402] An emotion engine built into the terminal recognizes the worker's emotional state. Inputs include the worker's voice, facial expressions, and behavioral data, which are analyzed to identify and output the emotional state. Specifically, it uses voice analysis and facial recognition technology to detect stress and anxiety.

[0403] Step 7:

[0404] The server analyzes emotional state data obtained from the emotion engine and adjusts how maintenance procedures are presented. The input is the recognized emotional data, and based on this, it adjusts and outputs support procedures tailored to the worker. Specifically, it adjusts the level of detail in explanations and adds alerts.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0408] [Third Embodiment]

[0409] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0410] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0421] The system of the present invention includes an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by operators. The roles and operations of each component will be described in detail below.

[0422] The server creates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This flight plan includes route information designed to allow the UAV to patrol the destination safely and efficiently, and also takes weather and terrain data into consideration. Based on this information, the server sends real-time instructions to the UAV.

[0423] The unmanned aerial vehicle (UAV) flies autonomously based on instructions from the server, patrolling the area around the communication device. During this time, it collects video and environmental data from the communication device using its equipped cameras and sensors. This collected data is transmitted to the server in real time.

[0424] The terminal is a device held by the worker and functions as an interface that displays analysis results and maintenance procedures from the server. The server analyzes the transmitted data using artificial intelligence to detect anomalies and identify their causes. The analysis results, along with the optimal maintenance procedures, are sent to the terminal, and the worker uses this information to repair or adjust the communication equipment.

[0425] As a concrete example, if a malfunction occurs in a communication device installed at a high altitude, the unmanned aerial vehicle (UAV) will patrol the area and send detailed video footage back to the server. Based on the video, the server will identify the abnormality, such as a damaged cable or a loose connection, and create specific procedures to correct it. Workers can then safely and efficiently perform repairs by checking the graphical procedures displayed using AR technology through a terminal.

[0426] This system enables the maintenance of communication equipment quickly and accurately, regardless of the operator's experience. The introduction of this system will enhance the reliability of the communication infrastructure and reduce the workload on technicians.

[0427] The following describes the processing flow.

[0428] Step 1:

[0429] Based on the location information of communication devices requiring patrol, the server generates a flight plan for the unmanned aerial vehicle (UAV), taking into account weather data and current operational conditions. This flight plan includes details of an efficient and safe patrol route.

[0430] Step 2:

[0431] The unmanned aerial vehicle (UAV) begins flight autonomously according to the flight plan received from the server via the terminal. During flight, it collects video data of the surroundings of the communication device using high-precision cameras and various sensors.

[0432] Step 3:

[0433] The server receives data transmitted in real time from the unmanned aerial vehicle. The server's artificial intelligence analysis engine processes this data in real time and uses image analysis algorithms to detect anomalies in the communication equipment.

[0434] Step 4:

[0435] Based on the detected anomaly, the server generates the optimal repair procedure by comparing it with past repair history. This repair procedure clearly specifies the particular work methods and tools to be used.

[0436] Step 5:

[0437] Repair instructions generated from the server are sent to the user's (worker's) terminal. The terminal displays the maintenance instructions and necessary information visually using augmented reality technology.

[0438] Step 6:

[0439] The user, acting as the operator, utilizes the information and AR guidance provided on the terminal to efficiently and safely repair and adjust communication equipment. This allows the operator to quickly resolve any problems.

[0440] (Example 1)

[0441] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0442] Communication equipment installed at high altitudes is difficult to inspect and maintain, requiring early detection and rapid response to any abnormalities. However, conventional methods are dependent on the skill level of the workers and weather conditions, making efficient maintenance challenging. Furthermore, there is a need for automated abnormality detection and the provision of efficient maintenance procedures.

[0443] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0444] In this invention, the server includes means for patrolling communication equipment using a data processing device and collecting information, evaluation means for analyzing the collected information using a machine learning model and detecting anomalies, means for generating and providing optimal maintenance actions to the operator based on the analysis results, and means for creating a flight plan using a machine learning model. This enables early detection of anomalies and the provision of efficient maintenance procedures, independent of the individual capabilities of the operator.

[0445] A "data processing device" is an autonomous, mobile device designed to patrol communication equipment installed at high altitudes and collect information about the surrounding environment.

[0446] "Information" refers to data about the surrounding environment and state of communication equipment, including images and audio acquired through cameras and sensors, as well as environmental measurements.

[0447] A "machine learning model" is a set of algorithms used to analyze large amounts of data, identify patterns, and detect anomalies, and includes artificial intelligence technology.

[0448] "Evaluation means" refers to a system process that uses a machine learning model based on collected information to determine the normality of communication equipment and identify abnormalities.

[0449] "Maintenance actions" refer to an action plan that outlines the work and processes necessary to correct malfunctions in communication equipment, and includes specific instructions for workers.

[0450] A "flight plan" is a plan that outlines the route information a data processing unit will take when patrolling communication equipment, and is designed to maximize safety and efficiency.

[0451] "Means to be provided to the worker" refers to technologies for appropriately presenting the analyzed results and maintenance actions to the worker, and includes visual display methods.

[0452] The embodiments for carrying out the present invention are described below.

[0453] This system includes a data processing unit (unmanned aerial vehicle), a computer system (server) for controlling it and analyzing data, and an operating device (terminal) used by operators. This allows for the rapid detection of abnormalities in communication equipment and enables appropriate maintenance.

[0454] The server generates a flight plan for the data processing unit to patrol the communication equipment. Specifically, the server uses a generation AI model to analyze real-time weather data and terrain information to design a safe and efficient flight path. Furthermore, the server sends commands to the data processing unit via a communication protocol. The MQTT protocol is used as needed.

[0455] The data processing unit autonomously patrols the area around the communication equipment according to instructions from the server, and returns information collected by its equipped cameras and sensors to the server in real time. This information includes image data, environmental data, and acoustic data.

[0456] The server uses the received information to detect anomalies with a machine learning model and generates maintenance actions based on the results. The evaluation method utilizes algorithms suitable for image analysis, such as convolutional neural networks (CNNs).

[0457] Subsequently, the server transmits the analyzed anomaly information and maintenance procedures to the terminal, which then visually presents this information to the worker. Specifically, augmented reality technology is used to display instructions in a format that is easy for the worker to understand.

[0458] For example, if an anomaly is detected in communication equipment installed at a high altitude, a data processing unit flies over the area and collects detailed video footage using a camera. The server analyzes this data, and if an anomaly is identified, it provides the worker with the most suitable repair procedure via a terminal.

[0459] An example of a prompt message for a generative AI model is, "Prepare a means to patrol communication devices and detect anomalies from the collected information."

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The server generates flight plans using a generative AI model. It receives real-time weather data and terrain information as input. This data is analyzed to calculate a safe and efficient route. As output, it creates a flight plan including route information and sends it to a data processing unit. Specifically, the server obtains wind speed and rainfall information via a weather API and performs route optimization based on this information.

[0463] Step 2:

[0464] The data processing unit patrols the area around the communication equipment according to the flight plan received from the server. It receives command routes from the server as input and transmits video and environmental data from the communication equipment to the server as output. Specific operations include acquiring detailed images and measurement data in real time during patrols using cameras and sensors.

[0465] Step 3:

[0466] The server analyzes video and environmental data received from the data processing unit. It receives data from multiple sensors as input and uses a machine learning model to detect anomalies. As output, it generates anomaly identification and diagnostic results, and plans maintenance actions based on these results. Specifically, the server utilizes a convolutional neural network (CNN) to identify anomalies in the video data.

[0467] Step 4:

[0468] The server transmits analysis results and maintenance actions to the terminal. Using the diagnostic results from the analysis as input, it generates specific instructions for the worker as output. These specific actions include creating documents containing repair procedures and important points to note.

[0469] Step 5:

[0470] The terminal provides information to the worker based on instructions from the server. It receives maintenance instructions from the server as input and displays the information in a visually understandable format as output. Specifically, it utilizes augmented reality (AR) technology to display repair locations as 3D models, thereby supporting the worker's understanding.

[0471] Step 6:

[0472] The user (worker) performs the actual maintenance work based on the information displayed on the terminal. The input is the instruction information from the terminal, and the output is the restoration of the communication equipment to normal operation. Specific actions include following the steps indicated by the terminal in order and reporting the completion of the work to the server upon completion.

[0473] (Application Example 1)

[0474] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] Traditional inspection and maintenance work on communication equipment involved working at heights, posing safety challenges and placing a heavy burden on workers. Furthermore, detecting anomalies and determining repair methods relied on the worker's experience, resulting in a lack of efficiency and consistency. This made rapid response difficult and posed a risk of compromising the reliability of the communication infrastructure.

[0476] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0477] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to workers based on the analysis results; and means for immediately notifying workers when an anomaly is detected and visually displaying repair and verification procedures using augmented reality technology. This enables safe and efficient patrolling and maintenance of communication equipment, reduces the burden on workers, and improves the reliability of the communication infrastructure.

[0478] An "unmanned aerial vehicle" is a device that flies remotely or autonomously and performs a specific function.

[0479] A "communication device" is an electronic device installed at a high location that transmits and receives data.

[0480] "Data" refers to information about the communication equipment and surrounding environment collected by the unmanned aerial vehicle.

[0481] "Artificial intelligence" is a technology in which computer programs mimic human intelligence, analyze data, and detect anomalies.

[0482] "Analysis means" refers to methods or mechanisms for evaluating data and determining whether or not there are abnormalities.

[0483] A "maintenance procedure" is a standard set of operations established to correct malfunctions in communication equipment.

[0484] Augmented reality technology is a technology that overlays virtual information onto the real world's field of view.

[0485] A "smartphone" is a portable information processing device that allows for communication and the use of various applications.

[0486] A "head-mounted display" is a device that displays images in the wearer's field of vision.

[0487] "Notification" refers to the act of communicating information to inform workers of the occurrence of an abnormality.

[0488] The present invention comprises an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by an operator. The following method is used to realize this system.

[0489] The server generates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. The server considers weather data and terrain information to create route information that enables autonomous flight, and sends real-time instructions to the UAV. In this process, GIS data is used to design a safe and efficient route.

[0490] The unmanned aerial vehicle (UAV) uses its onboard cameras and sensors to collect video data from communication devices and data about the surrounding environment. The collected data is immediately transmitted to a server. The server analyzes this data using artificial intelligence technology and analytical tools (e.g., TensorFlow and OpenCV) to detect anomalies. If an anomaly is detected, the server generates the optimal maintenance procedure based on the analysis results and presents it to the operator.

[0491] The terminals include smartphones and head-mounted displays. Maintenance procedures transmitted from the server are visually displayed on the terminal using augmented reality technology. This is achieved using Unity's ARFoundation, among other technologies. Workers can use this information to perform repair work quickly and safely. Furthermore, workers are immediately notified if any abnormalities are detected.

[0492] As a concrete example, consider the inspection of solar panels installed on the roof of a large warehouse. An unmanned aerial vehicle (UAV) patrols the area, identifying dirt and damaged parts of the panels. A server analyzes this information and sends a notification if an anomaly is detected. Workers can safely perform the task using a head-mounted display, following repair instructions displayed in augmented reality.

[0493] An example of a prompt message is: "Please describe a high-altitude inspection system using unmanned aerial vehicles. Consider its applications comprehensively and describe in detail how it contributes to safety and efficiency, especially its application to security services." This invention will enable even unskilled personnel to quickly perform maintenance on communication infrastructure, significantly reducing the burden on technicians.

[0494] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0495] Step 1:

[0496] The server receives weather data and terrain information as input. It analyzes this data to design a flight path for safe and efficient flight and generates a flight plan. The generated flight plan is then transmitted to the unmanned aerial vehicle (UAV).

[0497] Step 2:

[0498] The unmanned aerial vehicle receives the flight plan and begins autonomous flight. It collects ambient environmental data and video data from communication devices using cameras and sensors, and transmits this data to a server in real time.

[0499] Step 3:

[0500] The server takes received video and environmental data as input and performs data analysis using artificial intelligence technology. The AI ​​model responsible for anomaly detection uses, for example, TensorFlow or OpenCV to determine whether or not an anomaly is present. Based on the analysis results, the server generates the optimal maintenance procedure if an anomaly is found. This result is sent to the worker's terminal.

[0501] Step 4:

[0502] The terminal visually displays the analysis results and maintenance procedures received from the server using augmented reality technology. Workers can then review the maintenance procedures as an AR overlay via a head-mounted display or similar device and perform repair work. Using Unity's ARFoundation makes the operating procedures easier to understand visually.

[0503] Step 5:

[0504] Based on notification information displayed on the device, users can quickly identify the repair location and perform maintenance work. By referring to augmented reality visual information, even inexperienced users can perform maintenance work efficiently.

[0505] In this way, by following all steps, the unmanned aerial vehicle, the data analysis server, and the worker's terminal work in coordination, making maintenance work on communication equipment more efficient.

[0506] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0507] This invention relates to a system comprising an unmanned aerial vehicle, a server for data analysis, a terminal for use by an operator, and an emotion engine for recognizing user emotions. The roles and specific operations of each component are described below.

[0508] The server uses location information and environmental data from the communication device to formulate a flight plan for the unmanned aerial vehicle (UAV). This plan specifies a safe and efficient route while adapting to environmental conditions. The UAV performs autonomous flight according to the plan received from the server, collecting video data using sensors and cameras during patrols. This data is sent back to the server, where artificial intelligence is used to analyze for any anomalies.

[0509] The terminal provides the worker with optimal maintenance procedures generated based on the analyzed data. These procedures are also presented visually using augmented reality technology, allowing the worker to follow them more intuitively.

[0510] A distinctive feature of this invention is the incorporation of an emotion engine. The emotion engine is built into the terminal and analyzes the worker's voice, facial expressions, and behavioral patterns to recognize their current emotional state. The server receives this emotional data and adjusts the method of providing maintenance procedures and the content of support according to the worker's mental state.

[0511] For example, if the emotion engine detects that a worker is stressed, the terminal will display a more detailed procedural guide, and the server will add response time and confirmation alerts to assist the worker. In this way, creating an environment where workers can work with peace of mind improves the safety and efficiency of maintenance work.

[0512] This system reduces the risks faced by technicians and provides comprehensive support for fast and accurate maintenance work.

[0513] The following describes the processing flow.

[0514] Step 1:

[0515] The server creates a flight plan for the unmanned aerial vehicle (UAV) based on the location of the communication device and current environmental data. During this process, it considers weather conditions and terrain information to select a safe and efficient route. The generated flight plan is then transmitted to the UAV.

[0516] Step 2:

[0517] The unmanned aerial vehicle (UAV) begins flying autonomously according to the flight plan received from the server. During flight, it collects video data from its communication devices and other sensor data in real time and transmits this data to the server.

[0518] Step 3:

[0519] The server uses the received data to perform analysis using artificial intelligence. The AI ​​analyzes the image data and detects anomalies. For example, it identifies loose or damaged parts of cables and reports the anomaly.

[0520] Step 4:

[0521] The server generates the optimal maintenance procedure based on the analysis results. This procedure includes specific repair methods and the tools to be used. This information is then sent to the worker's terminal.

[0522] Step 5:

[0523] The user, acting as the worker, begins the task based on the information provided by the terminal. The terminal assists the worker by visually displaying the repair location and procedure using augmented reality technology.

[0524] Step 6:

[0525] The emotion engine built into the terminal analyzes the worker's voice and facial expressions to evaluate their emotional state in real time. For example, if anxiety or lack of concentration is detected, the terminal increases the amount of work support information.

[0526] Step 7:

[0527] The server receives feedback from the emotion engine and adjusts the way work procedures are presented and the level of detail in maintenance instructions. It provides appropriate support so that users can work with confidence.

[0528] Step 8:

[0529] The user completes the task with assistance from the emotion engine and the server. After completion, they submit a report from their terminal, and the server records the results in a database. This data will be used for future analysis and to improve the accuracy of the assistance.

[0530] (Example 2)

[0531] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0532] In the inspection of communication equipment at high altitudes using unmanned aerial vehicles, there is a need for anomaly detection and efficient maintenance work. However, there is a lack of support that takes into account the emotional state of the workers, and the safety and efficiency of maintenance work are not adequately ensured.

[0533] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0534] In this invention, the server includes means for collecting information using an unmanned aerial vehicle, means for analyzing the collected information with artificial intelligence and detecting anomalies, means for suggesting optimal maintenance work based on the analysis results, and means for recognizing the emotional state of the worker and adjusting the maintenance work based on that data. This enables improvements in the safety and efficiency of maintenance work, as well as mental support for the worker.

[0535] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and collects data using onboard sensors and cameras.

[0536] "Communication equipment" refers to devices installed at high altitudes for communication purposes, and is hardware used for sending and receiving data.

[0537] "Information" refers to data used for analysis, such as image data and environmental data collected by unmanned aerial vehicles.

[0538] "Artificial intelligence" is a technology that mimics the intelligent functions of computer systems, and in particular, it refers to programs that have the ability to detect anomalies based on data.

[0539] "Maintenance work" refers to tasks performed to maintain the functionality of communication equipment and to repair it, and is an action taken to efficiently resolve problems.

[0540] An "emotion engine" is a system that analyzes the worker's voice, facial expressions, etc., to recognize their emotional state at that time.

[0541] "Means" is a term that refers to a device, a group of devices, or a method used to achieve a specific function or purpose.

[0542] Augmented reality technology is a technique that overlays digital information onto the real world and is used to provide visual assistance to workers.

[0543] Modes for carrying out the invention

[0544] The present invention is a system that includes an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine for recognizing the user's emotions.

[0545] System Configuration

[0546] Server operation

[0547] The server receives and stores data transmitted from the unmanned aerial vehicle (UAV) and performs analysis using artificial intelligence. The software used includes generative AI models, which are employed for pattern recognition and anomaly detection in the data. It also accesses external APIs to obtain weather conditions and geographical data, and creates flight plans for the UAV. This plan identifies the optimal flight path and instructs the UAV accordingly.

[0548] Operation of unmanned aerial vehicles

[0549] The unmanned aerial vehicle (UAV) flies autonomously and collects data from communication equipment installed at high altitudes. The hardware used for collection includes high-resolution cameras and various sensors. Flight plans are provided by a server, and the UAV monitors the environment according to this plan and transmits the necessary information to the server.

[0550] Terminal operation

[0551] The terminal presents the worker with the most suitable maintenance tasks based on analysis results received from the server. Augmented reality technology is used for the presentation, enhancing visual assistance and enabling intuitive work procedures. In addition, a built-in emotion engine analyzes the worker's voice and facial expressions to assess their mental state. If the worker is experiencing stress, support is enhanced by providing more detailed explanations and appropriate alerts.

[0552] Specific example

[0553] For example, while a worker is inspecting an unmanned aerial vehicle, the server can detect anomalies in real time based on a generated AI model. The worker's terminal displays details of the anomaly and specific countermeasures via augmented reality. An emotion engine, sensing the worker's tension from their tone of voice, displays tips on the terminal to help them relax.

[0554] Example of a prompt

[0555] "Please explain how maintenance procedures can be dynamically adjusted based on the emotional state of the workers."

[0556] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0557] Step 1:

[0558] The server retrieves the latest environmental data from external weather APIs and geographic information databases. Inputs include location information and the types of environmental data required, and queries are executed against the databases and APIs. Outputs include environmental data necessary for developing safe flight plans for unmanned aerial vehicles (UAVs). This data is used to execute algorithms that determine the optimal flight route for the UAV. Specific operations include weather pattern analysis and terrain analysis.

[0559] Step 2:

[0560] The unmanned aerial vehicle (UAV) begins autonomous flight based on a flight plan received from a server. Inputs include the flight plan transmitted from the server and real-time environmental data collected by onboard sensors and cameras. During flight, the UAV periodically takes photos and videos and collects other environmental data with its sensors. Outputs include sending this data as data packets to the server. Specific operations include high-resolution video capture and temperature and humidity measurement.

[0561] Step 3:

[0562] The server receives data transmitted from the unmanned aerial vehicle (UAV) and performs analysis for anomaly detection using a generative AI model. Inputs include image data and environmental data collected by the UAV. The server inputs this data into an AI algorithm and compares it to known anomaly patterns. The output provides the presence or absence of anomalies and their details as analysis results. Specific operations include pattern matching utilizing image recognition technology.

[0563] Step 4:

[0564] The terminal receives analysis results from the server and generates maintenance procedures to present to the operator. Input includes a database of anomaly detection results and repair procedures provided by the server. The terminal uses augmented reality technology to visually present the procedures in a way that the operator can intuitively understand. Output provides the operator with visually guided maintenance procedures. Specific actions include a visual overlay using AR markers.

[0565] Step 5:

[0566] The terminal uses an emotion engine to analyze the worker's voice and facial expressions and assess their emotional state. Inputs include real-time audio data and video feeds. Based on the analysis results, the server adjusts work procedures and alerts as needed, and provides customized instructions to the worker as output. Specific actions include the dynamic generation of alerts and support messages to address different emotional states.

[0567] (Application Example 2)

[0568] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] In patrol systems using unmanned aerial vehicles, there is a need for means to accurately detect abnormalities in communication equipment and to ensure that workers can perform maintenance work with peace of mind. Furthermore, it is necessary to flexibly adjust maintenance procedures according to the emotional state of the workers to guarantee an efficient and safe working environment.

[0570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0571] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to the worker based on the analysis results; and means equipped with an emotion engine that recognizes the worker's emotional state and adjusts the presented maintenance procedures according to the worker's emotions. This enables anomaly detection of communication equipment, improved efficiency of maintenance work, and enhanced safety.

[0572] An "unmanned aerial vehicle" is an autonomous flying machine used to patrol communication equipment installed at high altitudes and efficiently collect data.

[0573] "Artificial intelligence that analyzes data and detects anomalies" is a computer program that uses collected data to identify irregular patterns and anomalies.

[0574] "A means of generating and presenting optimal maintenance procedures to workers" refers to a method of formulating the most efficient repair and maintenance methods based on analysis results and providing them to workers.

[0575] The "emotion engine" is an analytical device that grasps the psychological state of a worker from their facial expressions, voice, and movement data, and provides appropriate support according to that state.

[0576] Augmented reality technology is a technique that overlays digital data onto visual information from the real world, allowing workers to intuitively understand the information.

[0577] The system for carrying out this invention consists of an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine that recognizes the operator's emotions.

[0578] The server first develops a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This plan is refined using environmental data to provide a safe and efficient route. The UAV flies autonomously, collecting data using cameras and sensors during the patrol. This data is transmitted to the server, where artificial intelligence analyzes the data and detects any anomalies.

[0579] Based on the analysis results, the server generates the optimal maintenance procedure. This procedure is displayed on the worker's terminal and presented visually using augmented reality technology, making it easier for the worker to intuitively understand the procedure. The emotion engine built into the terminal recognizes the worker's emotional state based on voice, facial expressions, and behavioral data, and the server uses this emotional data to provide the maintenance procedure in a way that is appropriate for the worker. For example, if the worker is nervous, detailed guides and alerts are added to support the work.

[0580] Implementing this system will enable the detection of abnormalities in communication devices, provide workers with flexible maintenance procedures, and improve worker safety and work efficiency.

[0581] A concrete example is maintenance work during the installation of new equipment in a factory. In this scenario, an unmanned aerial vehicle (UAV) inspects the equipment and collects data. If a server detects an anomaly, it provides detailed instructions to the worker via a terminal. If the worker is stressed, an emotion engine detects this and provides additional support.

[0582] An example of a prompt for a generative AI model is, "Develop effective support strategies based on sentiment data as workers perform maintenance on new equipment."

[0583] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0584] Step 1:

[0585] The server develops flight plans for unmanned aerial vehicles (UAVs) to patrol communication equipment installed at high altitudes. Inputs include environmental data and historical flight data, and the server calculates and outputs the optimal route based on this data. Specifically, it automatically generates safe and efficient routes, taking into account environmental conditions such as wind speed and temperature.

[0586] Step 2:

[0587] The unmanned aerial vehicle (UAV) performs autonomous flight according to a flight plan sent from a server. Its input is flight plan data from the server; it uses this as a basis for actual flight, collecting images and sensor data. The collected data is then transferred back to the server as output. Specific operations include obstacle avoidance and real-time position adjustment during flight.

[0588] Step 3:

[0589] The server analyzes data sent from unmanned aerial vehicles. Inputs include image data and sensor data, which are analyzed using an artificial intelligence model to detect anomalies. Outputs are reports indicating the presence or absence of anomalies. For example, image processing techniques are used to check the surface condition of equipment.

[0590] Step 4:

[0591] The server generates the optimal maintenance procedure based on the analysis results and sends it to the terminal. The input is the anomaly detection result, and based on this, it formulates and outputs an appropriate maintenance procedure. Specifically, it extracts the necessary tasks from the equipment manual and generates a procedure document.

[0592] Step 5:

[0593] The terminal presents the received maintenance procedures to the worker using augmented reality technology. The input is maintenance procedure data from the server, which is then displayed and output as visual information. Specifically, guidelines are displayed in the worker's field of view via an AR headset.

[0594] Step 6:

[0595] An emotion engine built into the terminal recognizes the worker's emotional state. Inputs include the worker's voice, facial expressions, and behavioral data, which are analyzed to identify and output the emotional state. Specifically, it uses voice analysis and facial recognition technology to detect stress and anxiety.

[0596] Step 7:

[0597] The server analyzes emotional state data obtained from the emotion engine and adjusts how maintenance procedures are presented. The input is the recognized emotional data, and based on this, it adjusts and outputs support procedures tailored to the worker. Specifically, it adjusts the level of detail in explanations and adds alerts.

[0598] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0599] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0600] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0601] [Fourth Embodiment]

[0602] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0603] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0604] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0605] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0606] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0607] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0608] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0609] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0610] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0615] The system of the present invention includes an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by operators. The roles and operations of each component will be described in detail below.

[0616] The server creates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This flight plan includes route information designed to allow the UAV to patrol the destination safely and efficiently, and also takes weather and terrain data into consideration. Based on this information, the server sends real-time instructions to the UAV.

[0617] The unmanned aerial vehicle (UAV) flies autonomously based on instructions from the server, patrolling the area around the communication device. During this time, it collects video and environmental data from the communication device using its equipped cameras and sensors. This collected data is transmitted to the server in real time.

[0618] The terminal is a device held by the worker and functions as an interface that displays analysis results and maintenance procedures from the server. The server analyzes the transmitted data using artificial intelligence to detect anomalies and identify their causes. The analysis results, along with the optimal maintenance procedures, are sent to the terminal, and the worker uses this information to repair or adjust the communication equipment.

[0619] As a concrete example, if a malfunction occurs in a communication device installed at a high altitude, the unmanned aerial vehicle (UAV) will patrol the area and send detailed video footage back to the server. Based on the video, the server will identify the abnormality, such as a damaged cable or a loose connection, and create specific procedures to correct it. Workers can then safely and efficiently perform repairs by checking the graphical procedures displayed using AR technology through a terminal.

[0620] This system enables the maintenance of communication equipment quickly and accurately, regardless of the operator's experience. The introduction of this system will enhance the reliability of the communication infrastructure and reduce the workload on technicians.

[0621] The following describes the processing flow.

[0622] Step 1:

[0623] Based on the location information of communication devices requiring patrol, the server generates a flight plan for the unmanned aerial vehicle (UAV), taking into account weather data and current operational conditions. This flight plan includes details of an efficient and safe patrol route.

[0624] Step 2:

[0625] The unmanned aerial vehicle (UAV) begins flight autonomously according to the flight plan received from the server via the terminal. During flight, it collects video data of the surroundings of the communication device using high-precision cameras and various sensors.

[0626] Step 3:

[0627] The server receives data transmitted in real time from the unmanned aerial vehicle. The server's artificial intelligence analysis engine processes this data in real time and uses image analysis algorithms to detect anomalies in the communication equipment.

[0628] Step 4:

[0629] Based on the detected anomaly, the server generates the optimal repair procedure by comparing it with past repair history. This repair procedure clearly specifies the particular work methods and tools to be used.

[0630] Step 5:

[0631] Repair instructions generated from the server are sent to the user's (worker's) terminal. The terminal displays the maintenance instructions and necessary information visually using augmented reality technology.

[0632] Step 6:

[0633] The user, acting as the operator, utilizes the information and AR guidance provided on the terminal to efficiently and safely repair and adjust communication equipment. This allows the operator to quickly resolve any problems.

[0634] (Example 1)

[0635] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0636] Communication equipment installed at high altitudes is difficult to inspect and maintain, requiring early detection and rapid response to any abnormalities. However, conventional methods are dependent on the skill level of the workers and weather conditions, making efficient maintenance challenging. Furthermore, there is a need for automated abnormality detection and the provision of efficient maintenance procedures.

[0637] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0638] In this invention, the server includes means for patrolling communication equipment using a data processing device and collecting information, evaluation means for analyzing the collected information using a machine learning model and detecting anomalies, means for generating and providing optimal maintenance actions to the operator based on the analysis results, and means for creating a flight plan using a machine learning model. This enables early detection of anomalies and the provision of efficient maintenance procedures, independent of the individual capabilities of the operator.

[0639] A "data processing device" is an autonomous, mobile device designed to patrol communication equipment installed at high altitudes and collect information about the surrounding environment.

[0640] "Information" refers to data about the surrounding environment and state of communication equipment, including images and audio acquired through cameras and sensors, as well as environmental measurements.

[0641] A "machine learning model" is a set of algorithms used to analyze large amounts of data, identify patterns, and detect anomalies, and includes artificial intelligence technology.

[0642] "Evaluation means" refers to a system process that uses a machine learning model based on collected information to determine the normality of communication equipment and identify abnormalities.

[0643] "Maintenance actions" refer to an action plan that outlines the work and processes necessary to correct malfunctions in communication equipment, and includes specific instructions for workers.

[0644] A "flight plan" is a plan that outlines the route information a data processing unit will take when patrolling communication equipment, and is designed to maximize safety and efficiency.

[0645] "Means to be provided to the worker" refers to technologies for appropriately presenting the analyzed results and maintenance actions to the worker, and includes visual display methods.

[0646] The embodiments for carrying out the present invention are described below.

[0647] This system includes a data processing unit (unmanned aerial vehicle), a computer system (server) for controlling it and analyzing data, and an operating device (terminal) used by operators. This allows for the rapid detection of abnormalities in communication equipment and enables appropriate maintenance.

[0648] The server generates a flight plan for the data processing unit to patrol the communication equipment. Specifically, the server uses a generation AI model to analyze real-time weather data and terrain information to design a safe and efficient flight path. Furthermore, the server sends commands to the data processing unit via a communication protocol. The MQTT protocol is used as needed.

[0649] The data processing unit autonomously patrols the area around the communication equipment according to instructions from the server, and returns information collected by its equipped cameras and sensors to the server in real time. This information includes image data, environmental data, and acoustic data.

[0650] The server uses the received information to detect anomalies with a machine learning model and generates maintenance actions based on the results. The evaluation method utilizes algorithms suitable for image analysis, such as convolutional neural networks (CNNs).

[0651] Subsequently, the server transmits the analyzed anomaly information and maintenance procedures to the terminal, which then visually presents this information to the worker. Specifically, augmented reality technology is used to display instructions in a format that is easy for the worker to understand.

[0652] For example, if an anomaly is detected in communication equipment installed at a high altitude, a data processing unit flies over the area and collects detailed video footage using a camera. The server analyzes this data, and if an anomaly is identified, it provides the worker with the most suitable repair procedure via a terminal.

[0653] An example of a prompt message for a generative AI model is, "Prepare a means to patrol communication devices and detect anomalies from the collected information."

[0654] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0655] Step 1:

[0656] The server generates flight plans using a generative AI model. It receives real-time weather data and terrain information as input. This data is analyzed to calculate a safe and efficient route. As output, it creates a flight plan including route information and sends it to a data processing unit. Specifically, the server obtains wind speed and rainfall information via a weather API and performs route optimization based on this information.

[0657] Step 2:

[0658] The data processing unit patrols the area around the communication equipment according to the flight plan received from the server. It receives command routes from the server as input and transmits video and environmental data from the communication equipment to the server as output. Specific operations include acquiring detailed images and measurement data in real time during patrols using cameras and sensors.

[0659] Step 3:

[0660] The server analyzes video and environmental data received from the data processing unit. It receives data from multiple sensors as input and uses a machine learning model to detect anomalies. As output, it generates anomaly identification and diagnostic results, and plans maintenance actions based on these results. Specifically, the server utilizes a convolutional neural network (CNN) to identify anomalies in the video data.

[0661] Step 4:

[0662] The server transmits analysis results and maintenance actions to the terminal. Using the diagnostic results from the analysis as input, it generates specific instructions for the worker as output. These specific actions include creating documents containing repair procedures and important points to note.

[0663] Step 5:

[0664] The terminal provides information to the worker based on instructions from the server. It receives maintenance instructions from the server as input and displays the information in a visually understandable format as output. Specifically, it utilizes augmented reality (AR) technology to display repair locations as 3D models, thereby supporting the worker's understanding.

[0665] Step 6:

[0666] The user (worker) performs the actual maintenance work based on the information displayed on the terminal. The input is the instruction information from the terminal, and the output is the restoration of the communication equipment to normal operation. Specific actions include following the steps indicated by the terminal in order and reporting the completion of the work to the server upon completion.

[0667] (Application Example 1)

[0668] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] Traditional inspection and maintenance work on communication equipment involved working at heights, posing safety challenges and placing a heavy burden on workers. Furthermore, detecting anomalies and determining repair methods relied on the worker's experience, resulting in a lack of efficiency and consistency. This made rapid response difficult and posed a risk of compromising the reliability of the communication infrastructure.

[0670] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0671] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to workers based on the analysis results; and means for immediately notifying workers when an anomaly is detected and visually displaying repair and verification procedures using augmented reality technology. This enables safe and efficient patrolling and maintenance of communication equipment, reduces the burden on workers, and improves the reliability of the communication infrastructure.

[0672] An "unmanned aerial vehicle" is a device that flies remotely or autonomously and performs a specific function.

[0673] A "communication device" is an electronic device installed at a high location that transmits and receives data.

[0674] "Data" refers to information about the communication equipment and surrounding environment collected by the unmanned aerial vehicle.

[0675] "Artificial intelligence" is a technology in which computer programs mimic human intelligence, analyze data, and detect anomalies.

[0676] "Analysis means" refers to methods or mechanisms for evaluating data and determining whether or not there are abnormalities.

[0677] A "maintenance procedure" is a standard set of operations established to correct malfunctions in communication equipment.

[0678] Augmented reality technology is a technology that overlays virtual information onto the real world's field of view.

[0679] A "smartphone" is a portable information processing device that allows for communication and the use of various applications.

[0680] A "head-mounted display" is a device that displays images in the wearer's field of vision.

[0681] "Notification" refers to the act of communicating information to inform workers of the occurrence of an abnormality.

[0682] The present invention comprises an unmanned aerial vehicle, a server for analyzing data, and a terminal for use by an operator. The following method is used to realize this system.

[0683] The server generates a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. The server considers weather data and terrain information to create route information that enables autonomous flight, and sends real-time instructions to the UAV. In this process, GIS data is used to design a safe and efficient route.

[0684] The unmanned aerial vehicle (UAV) uses its onboard cameras and sensors to collect video data from communication devices and data about the surrounding environment. The collected data is immediately transmitted to a server. The server analyzes this data using artificial intelligence technology and analytical tools (e.g., TensorFlow and OpenCV) to detect anomalies. If an anomaly is detected, the server generates the optimal maintenance procedure based on the analysis results and presents it to the operator.

[0685] The terminals include smartphones and head-mounted displays. Maintenance procedures transmitted from the server are visually displayed on the terminal using augmented reality technology. This is achieved using Unity's ARFoundation, among other technologies. Workers can use this information to perform repair work quickly and safely. Furthermore, workers are immediately notified if any abnormalities are detected.

[0686] As a concrete example, consider the inspection of solar panels installed on the roof of a large warehouse. An unmanned aerial vehicle (UAV) patrols the area, identifying dirt and damaged parts of the panels. A server analyzes this information and sends a notification if an anomaly is detected. Workers can safely perform the task using a head-mounted display, following repair instructions displayed in augmented reality.

[0687] An example of a prompt message is: "Please describe a high-altitude inspection system using unmanned aerial vehicles. Consider its applications comprehensively and describe in detail how it contributes to safety and efficiency, especially its application to security services." This invention will enable even unskilled personnel to quickly perform maintenance on communication infrastructure, significantly reducing the burden on technicians.

[0688] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0689] Step 1:

[0690] The server receives weather data and terrain information as input. It analyzes this data to design a flight path for safe and efficient flight and generates a flight plan. The generated flight plan is then transmitted to the unmanned aerial vehicle (UAV).

[0691] Step 2:

[0692] The unmanned aerial vehicle receives the flight plan and begins autonomous flight. It collects ambient environmental data and video data from communication devices using cameras and sensors, and transmits this data to a server in real time.

[0693] Step 3:

[0694] The server takes received video and environmental data as input and performs data analysis using artificial intelligence technology. The AI ​​model responsible for anomaly detection uses, for example, TensorFlow or OpenCV to determine whether or not an anomaly is present. Based on the analysis results, the server generates the optimal maintenance procedure if an anomaly is found. This result is sent to the worker's terminal.

[0695] Step 4:

[0696] The terminal visually displays the analysis results and maintenance procedures received from the server using augmented reality technology. Workers can then review the maintenance procedures as an AR overlay via a head-mounted display or similar device and perform repair work. Using Unity's ARFoundation makes the operating procedures easier to understand visually.

[0697] Step 5:

[0698] Based on notification information displayed on the device, users can quickly identify the repair location and perform maintenance work. By referring to augmented reality visual information, even inexperienced users can perform maintenance work efficiently.

[0699] In this way, by following all steps, the unmanned aerial vehicle, the data analysis server, and the worker's terminal work in coordination, making maintenance work on communication equipment more efficient.

[0700] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0701] This invention relates to a system comprising an unmanned aerial vehicle, a server for data analysis, a terminal for use by an operator, and an emotion engine for recognizing user emotions. The roles and specific operations of each component are described below.

[0702] The server uses location information and environmental data from the communication device to formulate a flight plan for the unmanned aerial vehicle (UAV). This plan specifies a safe and efficient route while adapting to environmental conditions. The UAV performs autonomous flight according to the plan received from the server, collecting video data using sensors and cameras during patrols. This data is sent back to the server, where artificial intelligence is used to analyze for any anomalies.

[0703] The terminal provides the worker with optimal maintenance procedures generated based on the analyzed data. These procedures are also presented visually using augmented reality technology, allowing the worker to follow them more intuitively.

[0704] A distinctive feature of this invention is the incorporation of an emotion engine. The emotion engine is built into the terminal and analyzes the worker's voice, facial expressions, and behavioral patterns to recognize their current emotional state. The server receives this emotional data and adjusts the method of providing maintenance procedures and the content of support according to the worker's mental state.

[0705] For example, if the emotion engine detects that a worker is stressed, the terminal will display a more detailed procedural guide, and the server will add response time and confirmation alerts to assist the worker. In this way, creating an environment where workers can work with peace of mind improves the safety and efficiency of maintenance work.

[0706] This system reduces the risks faced by technicians and provides comprehensive support for fast and accurate maintenance work.

[0707] The following describes the processing flow.

[0708] Step 1:

[0709] The server creates a flight plan for the unmanned aerial vehicle (UAV) based on the location of the communication device and current environmental data. During this process, it considers weather conditions and terrain information to select a safe and efficient route. The generated flight plan is then transmitted to the UAV.

[0710] Step 2:

[0711] The unmanned aerial vehicle (UAV) begins flying autonomously according to the flight plan received from the server. During flight, it collects video data from its communication devices and other sensor data in real time and transmits this data to the server.

[0712] Step 3:

[0713] The server uses the received data to perform analysis using artificial intelligence. The AI ​​analyzes the image data and detects anomalies. For example, it identifies loose or damaged parts of cables and reports the anomaly.

[0714] Step 4:

[0715] The server generates the optimal maintenance procedure based on the analysis results. This procedure includes specific repair methods and the tools to be used. This information is then sent to the worker's terminal.

[0716] Step 5:

[0717] The user, acting as the worker, begins the task based on the information provided by the terminal. The terminal assists the worker by visually displaying the repair location and procedure using augmented reality technology.

[0718] Step 6:

[0719] The emotion engine built into the terminal analyzes the worker's voice and facial expressions to evaluate their emotional state in real time. For example, if anxiety or lack of concentration is detected, the terminal increases the amount of work support information.

[0720] Step 7:

[0721] The server receives feedback from the emotion engine and adjusts the way work procedures are presented and the level of detail in maintenance instructions. It provides appropriate support so that users can work with confidence.

[0722] Step 8:

[0723] The user completes the task with assistance from the emotion engine and the server. After completion, they submit a report from their terminal, and the server records the results in a database. This data will be used for future analysis and to improve the accuracy of the assistance.

[0724] (Example 2)

[0725] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0726] In the inspection of communication equipment at high altitudes using unmanned aerial vehicles, there is a need for anomaly detection and efficient maintenance work. However, there is a lack of support that takes into account the emotional state of the workers, and the safety and efficiency of maintenance work are not adequately ensured.

[0727] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0728] In this invention, the server includes means for collecting information using an unmanned aerial vehicle, means for analyzing the collected information with artificial intelligence and detecting anomalies, means for suggesting optimal maintenance work based on the analysis results, and means for recognizing the emotional state of the worker and adjusting the maintenance work based on that data. This enables improvements in the safety and efficiency of maintenance work, as well as mental support for the worker.

[0729] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and collects data using onboard sensors and cameras.

[0730] "Communication equipment" refers to devices installed at high altitudes for communication purposes, and is hardware used for sending and receiving data.

[0731] "Information" refers to data used for analysis, such as image data and environmental data collected by unmanned aerial vehicles.

[0732] "Artificial intelligence" is a technology that mimics the intelligent functions of computer systems, and in particular, it refers to programs that have the ability to detect anomalies based on data.

[0733] "Maintenance work" refers to tasks performed to maintain the functionality of communication equipment and to repair it, and is an action taken to efficiently resolve problems.

[0734] An "emotion engine" is a system that analyzes the worker's voice, facial expressions, etc., to recognize their emotional state at that time.

[0735] "Means" is a term that refers to a device, a group of devices, or a method used to achieve a specific function or purpose.

[0736] Augmented reality technology is a technique that overlays digital information onto the real world and is used to provide visual assistance to workers.

[0737] Modes for carrying out the invention

[0738] The present invention is a system that includes an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine for recognizing the user's emotions.

[0739] System Configuration

[0740] Server operation

[0741] The server receives and stores data transmitted from the unmanned aerial vehicle (UAV) and performs analysis using artificial intelligence. The software used includes generative AI models, which are employed for pattern recognition and anomaly detection in the data. It also accesses external APIs to obtain weather conditions and geographical data, and creates flight plans for the UAV. This plan identifies the optimal flight path and instructs the UAV accordingly.

[0742] Operation of unmanned aerial vehicles

[0743] The unmanned aerial vehicle (UAV) flies autonomously and collects data from communication equipment installed at high altitudes. The hardware used for collection includes high-resolution cameras and various sensors. Flight plans are provided by a server, and the UAV monitors the environment according to this plan and transmits the necessary information to the server.

[0744] Terminal operation

[0745] The terminal presents the worker with the most suitable maintenance tasks based on analysis results received from the server. Augmented reality technology is used for the presentation, enhancing visual assistance and enabling intuitive work procedures. In addition, a built-in emotion engine analyzes the worker's voice and facial expressions to assess their mental state. If the worker is experiencing stress, support is enhanced by providing more detailed explanations and appropriate alerts.

[0746] Specific example

[0747] For example, while a worker is inspecting an unmanned aerial vehicle, the server can detect anomalies in real time based on a generated AI model. The worker's terminal displays details of the anomaly and specific countermeasures via augmented reality. An emotion engine, sensing the worker's tension from their tone of voice, displays tips on the terminal to help them relax.

[0748] Example of a prompt

[0749] "Please explain how maintenance procedures can be dynamically adjusted based on the emotional state of the workers."

[0750] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0751] Step 1:

[0752] The server retrieves the latest environmental data from external weather APIs and geographic information databases. Inputs include location information and the types of environmental data required, and queries are executed against the databases and APIs. Outputs include environmental data necessary for developing safe flight plans for unmanned aerial vehicles (UAVs). This data is used to execute algorithms that determine the optimal flight route for the UAV. Specific operations include weather pattern analysis and terrain analysis.

[0753] Step 2:

[0754] The unmanned aerial vehicle (UAV) begins autonomous flight based on a flight plan received from a server. Inputs include the flight plan transmitted from the server and real-time environmental data collected by onboard sensors and cameras. During flight, the UAV periodically takes photos and videos and collects other environmental data with its sensors. Outputs include sending this data as data packets to the server. Specific operations include high-resolution video capture and temperature and humidity measurement.

[0755] Step 3:

[0756] The server receives data transmitted from the unmanned aerial vehicle (UAV) and performs analysis for anomaly detection using a generative AI model. Inputs include image data and environmental data collected by the UAV. The server inputs this data into an AI algorithm and compares it to known anomaly patterns. The output provides the presence or absence of anomalies and their details as analysis results. Specific operations include pattern matching utilizing image recognition technology.

[0757] Step 4:

[0758] The terminal receives analysis results from the server and generates maintenance procedures to present to the operator. Input includes a database of anomaly detection results and repair procedures provided by the server. The terminal uses augmented reality technology to visually present the procedures in a way that the operator can intuitively understand. Output provides the operator with visually guided maintenance procedures. Specific actions include a visual overlay using AR markers.

[0759] Step 5:

[0760] The terminal uses an emotion engine to analyze the worker's voice and facial expressions and assess their emotional state. Inputs include real-time audio data and video feeds. Based on the analysis results, the server adjusts work procedures and alerts as needed, and provides customized instructions to the worker as output. Specific actions include the dynamic generation of alerts and support messages to address different emotional states.

[0761] (Application Example 2)

[0762] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0763] In patrol systems using unmanned aerial vehicles, there is a need for means to accurately detect abnormalities in communication equipment and to ensure that workers can perform maintenance work with peace of mind. Furthermore, it is necessary to flexibly adjust maintenance procedures according to the emotional state of the workers to guarantee an efficient and safe working environment.

[0764] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0765] In this invention, the server includes means for patrolling communication equipment installed at high altitudes using an unmanned aerial vehicle and collecting data; means for analyzing the data and detecting anomalies using artificial intelligence; means for generating and presenting optimal maintenance procedures to the worker based on the analysis results; and means equipped with an emotion engine that recognizes the worker's emotional state and adjusts the presented maintenance procedures according to the worker's emotions. This enables anomaly detection of communication equipment, improved efficiency of maintenance work, and enhanced safety.

[0766] An "unmanned aerial vehicle" is an autonomous flying machine used to patrol communication equipment installed at high altitudes and efficiently collect data.

[0767] "Artificial intelligence that analyzes data and detects anomalies" is a computer program that uses collected data to identify irregular patterns and anomalies.

[0768] "A means of generating and presenting optimal maintenance procedures to workers" refers to a method of formulating the most efficient repair and maintenance methods based on analysis results and providing them to workers.

[0769] The "emotion engine" is an analytical device that grasps the psychological state of a worker from their facial expressions, voice, and movement data, and provides appropriate support according to that state.

[0770] Augmented reality technology is a technique that overlays digital data onto visual information from the real world, allowing workers to intuitively understand the information.

[0771] The system for carrying out this invention consists of an unmanned aerial vehicle, a server for data analysis, a terminal used by an operator, and an emotion engine that recognizes the operator's emotions.

[0772] The server first develops a flight plan for the unmanned aerial vehicle (UAV) to patrol communication equipment installed at high altitudes. This plan is refined using environmental data to provide a safe and efficient route. The UAV flies autonomously, collecting data using cameras and sensors during the patrol. This data is transmitted to the server, where artificial intelligence analyzes the data and detects any anomalies.

[0773] Based on the analysis results, the server generates the optimal maintenance procedure. This procedure is displayed on the worker's terminal and presented visually using augmented reality technology, making it easier for the worker to intuitively understand the procedure. The emotion engine built into the terminal recognizes the worker's emotional state based on voice, facial expressions, and behavioral data, and the server uses this emotional data to provide the maintenance procedure in a way that is appropriate for the worker. For example, if the worker is nervous, detailed guides and alerts are added to support the work.

[0774] Implementing this system will enable the detection of abnormalities in communication devices, provide workers with flexible maintenance procedures, and improve worker safety and work efficiency.

[0775] A concrete example is maintenance work during the installation of new equipment in a factory. In this scenario, an unmanned aerial vehicle (UAV) inspects the equipment and collects data. If a server detects an anomaly, it provides detailed instructions to the worker via a terminal. If the worker is stressed, an emotion engine detects this and provides additional support.

[0776] An example of a prompt for a generative AI model is, "Develop effective support strategies based on sentiment data as workers perform maintenance on new equipment."

[0777] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0778] Step 1:

[0779] The server develops flight plans for unmanned aerial vehicles (UAVs) to patrol communication equipment installed at high altitudes. Inputs include environmental data and historical flight data, and the server calculates and outputs the optimal route based on this data. Specifically, it automatically generates safe and efficient routes, taking into account environmental conditions such as wind speed and temperature.

[0780] Step 2:

[0781] The unmanned aerial vehicle (UAV) performs autonomous flight according to a flight plan sent from a server. Its input is flight plan data from the server; it uses this as a basis for actual flight, collecting images and sensor data. The collected data is then transferred back to the server as output. Specific operations include obstacle avoidance and real-time position adjustment during flight.

[0782] Step 3:

[0783] The server analyzes data sent from unmanned aerial vehicles. Inputs include image data and sensor data, which are analyzed using an artificial intelligence model to detect anomalies. Outputs are reports indicating the presence or absence of anomalies. For example, image processing techniques are used to check the surface condition of equipment.

[0784] Step 4:

[0785] The server generates the optimal maintenance procedure based on the analysis results and sends it to the terminal. The input is the anomaly detection result, and based on this, it formulates and outputs an appropriate maintenance procedure. Specifically, it extracts the necessary tasks from the equipment manual and generates a procedure document.

[0786] Step 5:

[0787] The terminal presents the received maintenance procedures to the worker using augmented reality technology. The input is maintenance procedure data from the server, which is then displayed and output as visual information. Specifically, guidelines are displayed in the worker's field of view via an AR headset.

[0788] Step 6:

[0789] An emotion engine built into the terminal recognizes the worker's emotional state. Inputs include the worker's voice, facial expressions, and behavioral data, which are analyzed to identify and output the emotional state. Specifically, it uses voice analysis and facial recognition technology to detect stress and anxiety.

[0790] Step 7:

[0791] The server analyzes emotional state data obtained from the emotion engine and adjusts how maintenance procedures are presented. The input is the recognized emotional data, and based on this, it adjusts and outputs support procedures tailored to the worker. Specifically, it adjusts the level of detail in explanations and adds alerts.

[0792] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0793] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0794] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0795] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0796] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0797] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0798] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0799] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0800] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0801] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0802] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0803] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0804] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0805] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0806] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0807] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0808] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0809] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0810] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0811] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0812] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0813] The following is further disclosed regarding the embodiments described above.

[0814] (Claim 1)

[0815] A means of using unmanned aerial vehicles to patrol communication equipment installed at high altitudes and collect data,

[0816] An analysis means using artificial intelligence to analyze the aforementioned data and detect anomalies,

[0817] A means to generate and present optimal maintenance procedures to the operator based on the analysis results,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, wherein the unmanned aerial vehicle includes means for adjusting the flight path using environmental data in order to safely fly along a pre-planned flight path.

[0821] (Claim 3)

[0822] The system according to claim 1, wherein the presentation to the worker is performed visually using augmented reality technology.

[0823] "Example 1"

[0824] (Claim 1)

[0825] A means of patrolling communication equipment installed at high places using a data processing device and collecting information,

[0826] An evaluation means that uses a machine learning model to analyze the aforementioned information and detect anomalies,

[0827] A means for generating and providing optimal maintenance actions to workers based on evaluation results,

[0828] A method for creating a flight plan using a machine learning model,

[0829] A system that includes this.

[0830] (Claim 2)

[0831] The system according to claim 1, wherein the data processing device includes means for adjusting the route using environmental information in order to safely patrol a pre-set travel route.

[0832] (Claim 3)

[0833] The system according to claim 1, wherein the provision to the worker is performed by a visual display using augmented reality technology.

[0834] "Application Example 1"

[0835] (Claim 1)

[0836] A means of using unmanned aerial vehicles to patrol communication equipment installed at high altitudes and collect data,

[0837] An analysis means using artificial intelligence to analyze the aforementioned data and detect anomalies,

[0838] A means to generate and present optimal maintenance procedures to the operator based on the analysis results,

[0839] A means of immediately notifying when an anomaly is detected and visually displaying repair and verification procedures using augmented reality technology,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, wherein the unmanned aerial vehicle includes means for adjusting the flight path using environmental data in order to safely fly along a pre-planned flight path.

[0843] (Claim 3)

[0844] The system according to claim 1, wherein the information is presented to the worker via a smartphone or head-mounted display.

[0845] "Example 2 of combining an emotion engine"

[0846] (Claim 1)

[0847] A device that uses unmanned aerial vehicles to patrol and collect information on communication equipment installed at high altitudes,

[0848] An analysis device using artificial intelligence to analyze the aforementioned information and detect anomalies,

[0849] A device that generates and presents optimal maintenance procedures to the operator based on the analysis results,

[0850] A device including an emotion engine that analyzes the worker's voice and facial expressions to recognize their current emotional state,

[0851] A device that adjusts the method of providing maintenance work based on worker emotion data,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, wherein the unmanned aerial vehicle is equipped with a device that adjusts its path using environmental information in order to safely fly a pre-planned flight route.

[0855] (Claim 3)

[0856] The system according to claim 1, wherein the presentation to the worker is performed visually using augmented reality technology.

[0857] "Application example 2 of combining emotional engines"

[0858] (Claim 1)

[0859] A means of using unmanned aerial vehicles to patrol communication equipment installed at high altitudes and collect data,

[0860] An analysis means using artificial intelligence to analyze the aforementioned data and detect anomalies,

[0861] A means to generate and present optimal maintenance procedures to the operator based on the analysis results,

[0862] A means equipped with an emotion engine that recognizes the emotional state of the worker and adjusts the presented maintenance procedures according to the worker's emotions,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, wherein the unmanned aerial vehicle includes means for adjusting the flight path using environmental data in order to safely fly along a pre-planned flight path.

[0866] (Claim 3)

[0867] The system according to claim 1, wherein the presentation to the worker is performed visually using augmented reality technology, and the content is further optimized according to the worker's emotional state. [Explanation of Symbols]

[0868] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of using unmanned aerial vehicles to patrol communication equipment installed at high altitudes and collect data, An analysis means using artificial intelligence to analyze the aforementioned data and detect anomalies, A means to generate and present optimal maintenance procedures to the operator based on the analysis results, A system that includes this.

2. The system according to claim 1, wherein the unmanned aerial vehicle is equipped with means for adjusting its flight path using environmental data in order to safely fly along a pre-planned flight path.

3. The system according to claim 1, wherein the presentation to the worker is performed visually using augmented reality technology.

Images