system

Abstract

We provide the system. [Solution] A means for collecting omnidirectional video data while moving within a fluid path, Means for collecting vibration and sound data within the fluid path, A data analysis means including a learning model that synthesizes and analyzes the aforementioned omnidirectional video data, vibration data, and audio data, A means for detecting anomalies based on the aforementioned analysis results and predicting signs of anomaly occurrence, A means for transmitting analysis data in real time via communication with a ground base station, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In inspections and maintenance within a fluid path such as a water supply and drainage system, not only are conventional manual visual inspections and regular maintenance inefficient, but it is also difficult to detect abnormalities promptly. For this reason, major problems such as water leakage and damage are often not addressed until they occur, which often takes a lot of cost and time. Furthermore, these operations usually rely heavily on manual labor, and there is a problem that a lot of labor and resources are required.

Means for Solving the Problems

[0005] To address this challenge, the present invention provides a system that combines means for collecting omnidirectional video data with means for collecting vibration and audio data, in a device that can be deployed within a fluid path. This enables the device to collect comprehensive data while moving through the fluid. Furthermore, it is equipped with an advanced learning model that integrates and analyzes this data, detecting abnormal patterns in real time and predicting future anomalies. In addition, a user interface is provided that displays the analysis results together with map information, enabling prompt maintenance and optimizing time and resources.

[0006] A "fluid pathway" is a pipe or tube provided for the movement of fluids such as liquids or gases.

[0007] "Omnidirectional video data" refers to a collection of video information that covers all directions (360 degrees) from a given point.

[0008] "Vibration data" refers to information that captures how the movement and displacement of an object change over time.

[0009] "Audio data" refers to data that records sound vibrations in digital or analog format.

[0010] A "learning model" refers to an algorithm or mathematical framework used to learn patterns and rules from data and perform predictions and classifications.

[0011] "Data analysis means" refers to technologies that provide processes and methods for integrating and analyzing collected data.

[0012] A "ground base station" is a facility or device installed on the ground for connecting with remote devices via wireless communication and transmitting and receiving data.

[0013] A "user interface" is an interface that provides information and functions of a system to the user in a visual or interactive form. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of the present invention consists of a specially designed device and its control software, as well as a server program and user interface for data analysis, in order to efficiently detect anomalies in a fluid path. As an embodiment of this invention, the specific processing details of the program are described below in natural language.

[0036] terminal

[0037] First, the terminal controls a device placed within the fluid path. This device is equipped with a camera that captures omnidirectional video and various sensors that collect vibration and audio data. The terminal controls the device to move at an appropriate speed within the fluid path and simultaneously issues commands to capture data at regular intervals. This allows the data to be packaged and transmitted to the base station in real time.

[0038] server

[0039] The server receives data sent from the ground base station and prepares it for data analysis. The server program integrates this data and uses a specific learning model to identify patterns of abnormal operation. For example, if the server detects an abnormal sound, it analyzes its frequency components and predicts the need for future repairs based on the patterns identified as abnormal, and notifies the user accordingly.

[0040] User

[0041] Users can review the analysis results in detail through the provided user interface. The interface plots anomalies on a map of the fluid path and displays detailed information about each anomaly, helping users respond quickly and accurately. Based on this information, users can decide whether to dispatch a maintenance team or take countermeasures, and contribute to improving the accuracy of the system by providing feedback.

[0042] This embodiment of the invention enables significantly more efficient and precise monitoring and maintenance of fluid pathways than conventional methods.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The terminal activates the devices placed within the fluid path. Once the devices are activated, the omnidirectional cameras and various sensors begin operating and perform initial calibration. After calibration is complete, the devices begin moving along the configured path at a constant speed.

[0046] Step 2:

[0047] The terminal instructs the moving device to periodically collect omnidirectional video and sensor data. The device generates data in packages at specified time intervals, making them available for the terminal to receive. These packages contain video, vibration, and audio data along with timestamps.

[0048] Step 3:

[0049] The terminal transmits the collected data to a ground base station in real time. The data is transmitted wirelessly and received by a server. The terminal also adds location information to each data package, allowing the collection point of each data to be identified.

[0050] Step 4:

[0051] The server receives data packages transmitted from ground base stations, verifies their integrity, and then prepares them for analysis. First, the server filters and denoises the data to improve processing efficiency, and then converts it into a format suitable for the learning model.

[0052] Step 5:

[0053] The server inputs the adjusted data into a learning model and executes the anomaly detection process. Specifically, it detects anomaly patterns and identifies predicted risks. Once the analysis results are obtained, they are compiled and saved as a report.

[0054] Step 6:

[0055] Users access analysis reports generated from the server through a user interface. The interface displays each anomaly on a map and presents detailed information about the detected locations. This allows users to take immediate corrective action.

[0056] Step 7:

[0057] Users decide on actions to take to address any anomalies they discover, such as dispatching a maintenance team as needed. Users can also feed back field-tested data to the server, which contributes to improving the overall accuracy of the system.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Early detection of anomalies in fluid pathways is crucial, but current technologies often delay detection, potentially leading to serious problems. Furthermore, when anomalies are detected, accurate location information and detailed analysis results are necessary, requiring efficient and sophisticated technology. Therefore, there is a need for the development of new systems that enable rapid and accurate anomaly detection, prediction, and information provision.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for integrating and analyzing omnidirectional video information, vibration information, and audio information within a fluid path; means for detecting anomalies and predicting signs of anomaly occurrence based on the analysis results; means for transmitting analysis information in real time via a communication network; and means for providing a human-machine interface for visualizing the analyzed information and identifying the location of anomalies. This enables rapid and accurate anomaly detection and prediction, as well as effective response through the provision of more detailed information.

[0063] A "fluid pathway" refers to a passage through which a liquid or gas moves, and includes pipelines and ducts.

[0064] "Omnidirectional video information" refers to video data that can capture the entire 360-degree field of view, encompassing all visual information of the surroundings from a specific location.

[0065] "Vibration information" refers to data on the physical vibrations of objects and structures, and is information on periodic motion measured by sensors.

[0066] "Audio information" refers to data that describes the characteristics and properties of sound, and is obtained through the capture of sound signals by sensors.

[0067] An "information analysis device" refers to a device that receives various types of information as input, integrates and analyzes them, and outputs the results.

[0068] A "learning algorithm" refers to a method and procedure for learning patterns based on input data and performing analysis or prediction.

[0069] A "human-machine interface" is a means and interface for a user to interact with a machine, and includes elements that enable the visualization and input of information.

[0070] A "communication network" refers to a system for transmitting and sharing information, and includes means for connecting computers and other devices.

[0071] "Anomaly detection" refers to the process or ability to identify a state that deviates from normal operation.

[0072] "Predictive forecasting" refers to analysis and prediction aimed at foreseeing potential future changes or anomalies.

[0073] The system of this invention is designed to efficiently detect anomalies in fluid pathways. Specific embodiments of each component are shown below.

[0074] terminal

[0075] The terminal controls sensor-equipped devices placed within the fluid path. These devices are equipped with cameras for capturing omnidirectional video information and various sensors for acquiring vibration and audio information. The terminal captures and packages data in real time as the devices move within the fluid path. The collected data is transmitted to a ground base station via a communication network. Typical communication technologies used include Wi-Fi and mobile data communication.

[0076] server

[0077] The server is a device that receives and analyzes data transmitted through ground base stations. The server is equipped with a learning algorithm that identifies anomaly patterns based on the integrated data. This allows the server to perform anomaly detection in real time. For example, generative AI models are used to recognize abnormal vibrations and sound patterns during the analysis. If an anomaly is detected, the server predicts its precursors and notifies the user of the necessary information.

[0078] User

[0079] Users can review the system's analysis results and obtain detailed information about anomalies. Through a dedicated human-machine interface, users can access detailed information, including data visualizations, enabling rapid response. The interface allows users to issue maintenance instructions and arrange for repairs to anomalies. Furthermore, the feedback received can be used to improve the system.

[0080] For example, if an abnormal sound pattern is detected within the fluid path, the location will be displayed in red on the map, and detailed frequency analysis data will be provided. Furthermore, information will be provided to the user through prompt messages such as, "Analyze the sound and vibration data collected by devices installed within the fluid path, and notify us if any abnormalities are found. Display the location of the abnormality on the map so that maintenance can be arranged as needed."

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The terminal activates sensor-equipped devices placed within the fluid path to collect omnidirectional video, vibration, and audio information. The input here is raw data from each sensor, which the terminal converts into a digital format and packages. This data package is transmitted to a ground base station via Wi-Fi or a mobile data network. Specifically, the device captures data every second as it moves and transmits it at regular time intervals.

[0084] Step 2:

[0085] The server decompresses the data packages received from the ground base station and verifies the data's integrity. The input is packaged data transmitted from the terminal, and the output is integrated data in a format suitable for analysis. The server sorts this data based on a time axis and prepares it for anomaly detection. During this process, it inserts an operation to perform predictive imputation if there is missing data.

[0086] Step 3:

[0087] The server performs analysis using a generative AI model with the integrated data. The input here is the prepared data obtained in step 2, and the output is the anomaly detection and prediction results. The server uses a learning algorithm to extract features from the data and identify anomaly patterns. If an anomaly is recognized, it predicts future anomaly occurrences based on those features. For example, it analyzes the frequency components of audio data and compares them with past anomaly cases.

[0088] Step 4:

[0089] The server converts the analysis results into a format understandable to the user and presents them through a dedicated human-machine interface. The input is the analysis results obtained in step 3, and the output is visualized information and specific recommendations. For example, it can plot anomalies on a fluid path map and display the specific causes of the anomalies and suggested countermeasures.

[0090] Step 5:

[0091] Users make quick and accurate decisions based on information obtained through the interface. The input is analytical information provided by the server, which users evaluate and issue maintenance instructions. Providing feedback also contributes to further system improvements. Specific actions include clicking on anomalies to view detailed information and sending instructions to the maintenance team via email.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] It is crucial to detect abnormalities in fluid pathways early and prevent unexpected failures of machinery and equipment. However, predicting abnormalities is currently difficult, requiring rapid and precise monitoring and judgment. Furthermore, there is a lack of visual information to smoothly identify abnormal locations and instruct corrective actions.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes information analysis means that include a model for integrating and analyzing omnidirectional visual data, vibration data, and acoustic data; means for detecting abnormalities and predicting signs of anomaly occurrence; means for transmitting sequential information via wireless communication; and means for visualizing the results of anomaly detection based on time and location information. This enables early detection of anomalies in the fluid path, accurate location identification, and rapid implementation of countermeasures.

[0097] A "device that moves within a fluid passage" is a device that moves within a fluid passage while acquiring information about its surroundings.

[0098] "Omnidirectional visual data" refers to data that includes video information of the entire surrounding area at a given point.

[0099] "Vibration data" refers to data that includes information about vibrations within a fluid passage.

[0100] "Acoustic data" refers to data that contains information related to sounds generated within a fluid passage.

[0101] An "information analysis device" is a device designed to analyze collected data.

[0102] A "model" is a computational method or algorithm used to identify a specific pattern.

[0103] "Abnormal" is a term that refers to an abnormal state that deviates from the normal range.

[0104] "Sequential information" is a term that refers to data that is collected and transmitted immediately.

[0105] "Wireless communication" is a method of transmitting information over long distances using radio waves.

[0106] "Visualization" refers to displaying data and information in a way that is easy for people to understand.

[0107] "Time and location information" refers to data that includes information about a geographical location at a specific point in time.

[0108] The system implementing this invention provides a comprehensive function for monitoring fluid pathways within a factory. Its elements and operating processes are described below.

[0109] The terminal moves within the fluid path and acquires data through multi-directionally positioned visual and acoustic sensors. These sensors are used to collect omnidirectional visual data, vibration data, and acoustic data. This data is continuously transmitted from the terminal to the server via wireless communication.

[0110] The server is the central device that receives the collected data and performs information analysis. The server has an information analysis system built in a Python environment and is equipped with an advanced computational model using TENSORFLOW®. This model identifies anomaly patterns and executes algorithms to predict the likelihood of anomaly occurrence early on. Furthermore, if an anomaly is detected, it identifies its location based on the data and performs visualization processing based on time and location information.

[0111] Users can view detailed analysis results through an administration screen designed with React.js. This user interface clearly displays the location of detected anomalies on a map, helping users quickly identify problems and take appropriate action.

[0112] As a concrete example, when introducing new machinery to a factory, this system can be used to monitor the fluid pathways, allowing for the rapid detection of problems that tend to occur in the initial stages of implementation, thereby preventing major failures. By utilizing a generative AI model, for example, by inputting a prompt such as, "If abnormal vibrations or sounds are detected in the fluid pathways of newly introduced machinery, please list their locations and countermeasures," it is possible to compare them with known abnormality patterns and suggest appropriate countermeasures.

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The terminal collects omnidirectional visual data, vibration data, and acoustic data from multidirectional sensors installed within the fluid channel. The input is real-world environmental information within the fluid channel, and the output is digital data that packages this data together. This data is acquired directly from the sensor devices and processed in real time.

[0116] Step 2:

[0117] The terminal transmits the collected digital data to the server via wireless communication. The input is the digital data packaged in step 1, and the output is the data received by the server. Specifically, the Wi-Fi module manages the data transmission and applies the appropriate protocol to prevent data loss.

[0118] Step 3:

[0119] The server feeds the received digital data into an analysis model to identify anomalous patterns. The input is the digital data received by the server, and the output is the analysis results of the patterns and signs of anomalies that have been determined to be anomalous. TensorFlow is used for this analysis, and the specific computational processes for anomaly detection are performed using machine learning algorithms.

[0120] Step 4:

[0121] The server identifies the location of the anomaly based on the analysis results, adds time and location information, and generates visualization data. The input is the analysis results obtained in step 3, and the output is the visualized anomaly information. Specifically, it integrates map information and analysis results, and highlights the necessary information using a visualization algorithm.

[0122] Step 5:

[0123] Users review visualized anomaly information through the management screen and determine the necessary countermeasures. The input is the visualized data provided in step 4, and the output is the countermeasures and action plan decided by the user. Specifically, the user interface effectively indicates the location of the anomaly and assists the user in making quick decisions.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] The present invention provides an advanced infrastructure management system that, in addition to monitoring fluid pathways and detecting anomalies, also has the ability to recognize the emotional state of the user. An embodiment of the system is configured as follows:

[0126] terminal

[0127] The terminal controls a device installed within the fluid path. This device has a set of sensors to capture omnidirectional video and collect vibration and audio data. The device moves along the fluid path, packaging the data collected by the terminal and transmitting it in real time to a ground base station.

[0128] server

[0129] The server processes data received from the base station. The received dataset undergoes predetermined noise reduction and data normalization before being analyzed by a learning model. Based on the analysis results, the server quickly detects anomalies, predicts potential risks, and takes appropriate action. Furthermore, this server program incorporates an emotion engine to evaluate user reactions based on the analysis results.

[0130] User

[0131] Users can view anomaly detection reports in real time through a user interface that displays data analysis information. The user interface visually shows detected anomalies on a map and provides detailed reports. Furthermore, the emotion engine analyzes the user's voice and facial expressions to automatically select the optimal information presentation method. This plays a role in increasing efficiency in the user's daily work and supporting accurate maintenance tasks.

[0132] As a concrete example, when users encounter complex anomaly analysis results, the emotion engine summarizes and presents the information to reduce their stress levels. For instance, if the urgency of an anomaly is judged to be low, the notification can be presented in simple language to avoid confusing the user.

[0133] This embodiment enables the system to adapt to environmental changes, improve the user experience, and achieve secure and efficient infrastructure management.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] The terminal activates the device placed within the fluid path and prepares to capture omnidirectional video. After activation, the device uses its omnidirectional camera and various sensors to collect video, vibration, and audio data as it moves along the designated route.

[0137] Step 2:

[0138] The terminal packages the collected data in real time and then transmits it to a ground base station. The package includes all the collected data as well as location information, which allows the data collection point to be identified.

[0139] Step 3:

[0140] The server receives data from the ground base station and preprocesses it by performing noise reduction and data filtering. The preprocessed data is then fed into a learning model to perform anomaly detection and risk prediction.

[0141] Step 4:

[0142] The server generates an anomaly report based on the analysis results. The report includes the type and location of the detected anomaly, as well as risk information that requires immediate action if necessary. Furthermore, the server uses an emotion engine to analyze the user's current emotional data in order to recognize the user's emotional state.

[0143] Step 5:

[0144] After the emotion engine evaluates the user's emotional state, the server uses the evaluation results to notify the user of an anomaly in the most appropriate way. For example, if the user is experiencing stress, the server will respond by presenting the information in a simplified format.

[0145] Step 6:

[0146] Users view analysis reports from the server through a user interface. The interface displays anomalies on a map, and detailed information for each anomaly is provided in an easily accessible format.

[0147] Step 7:

[0148] Users plan appropriate actions based on the report. Specifically, they may dispatch a maintenance team or arrange for additional inspection work. They can also contribute to continuous improvement by providing feedback to the system.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] In modern infrastructure, early detection of anomalies and rapid and effective information provision to users are crucial. However, conventional systems lacked sufficient accuracy in data noise reduction, assessment of anomaly urgency, and appropriate information presentation methods tailored to users' emotional states. As a result, on-site responses were delayed, leading to increased user stress. This invention aims to solve these problems and support safe and efficient infrastructure operation.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for denoising and normalizing data, means for analyzing anomalies using a learning model and evaluating their urgency, and means for evaluating the user's emotional state using sentiment analysis technology and providing optimal information. This enables real-time anomaly detection and the provision of appropriate information to reduce user stress.

[0154] A "fluid pathway" is a passage or pipe through which a fluid, such as a liquid or gas, moves.

[0155] "Video data" refers to digital data containing visual information acquired using devices such as cameras.

[0156] "Vibration data" refers to information about vibrations of objects or the environment that are recorded using sensors.

[0157] "Acoustic data" refers to data collected by sound sensors that includes information about the surrounding sounds.

[0158] "Data processing means" refers to the system's functions, including the process of removing noise and normalizing acquired data.

[0159] A "learning model" is a model that analyzes data based on machine learning algorithms to perform pattern recognition and anomaly detection.

[0160] "Anomaly detection" is the process of identifying unusual data patterns or behaviors that indicate potential problems in a system.

[0161] "Emotional analysis technology" is a technology that analyzes voice and facial expression data to evaluate a user's emotional state.

[0162] A "user interface" refers to the screens and devices that a user uses to interact with a system and obtain information.

[0163] "Optimal information presentation" means providing information in a format that is easy to understand and reduces stress, tailored to the user's current emotional state.

[0164] The present invention is an advanced infrastructure management technology aimed at early detection of anomalies in fluid pathways and providing users with appropriate information. This system collects data using a group of devices installed in the fluid pathway, analyzes it using a server, and provides an interface for users to view the information.

[0165] terminal

[0166] The terminal moves along a fluid path and controls multiple sensors. Specifically, it is equipped with a camera that captures omnidirectional video, an accelerometer that detects vibrations, and a microphone that picks up sound. The terminal compresses and packages the data obtained from these sensors in real time and transmits it to the base station. A dedicated communication module is used for data transmission, enabling real-time communication.

[0167] server

[0168] The server receives data transmitted from the base station and first performs noise reduction. It cleanses and normalizes the data using programming libraries such as Python and NumPy. Next, it runs an analysis model on the processed data using machine learning libraries such as TensorFlow to detect anomalies and assess their urgency. Furthermore, it integrates the analysis results with sentiment analysis technology to evaluate the user's emotional state and provide appropriate information.

[0169] User

[0170] Users can view these analysis results through a user interface. This interface operates on a web browser and is designed to allow users to intuitively understand the anomalies and their details. Based on sentiment analysis technology, the format of information presentation is adjusted according to the user's current emotional state, playing a role in reducing stress and confusion.

[0171] As a concrete example, consider a case where abnormal vibrations are detected in a part of the fluid path. Based on this data, the server quickly identifies the anomaly and provides the user with a notification such as, "Abnormal vibration detected: Vibration confirmed in fluid path sector 3. No immediate action is required," thereby supporting the user in responding calmly.

[0172] The following are some possible prompt statements for a generative AI model.

[0173] "Summarize the latest anomaly analysis results in fluid pathways and propose an appropriate information presentation method to reduce user stress."

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The device moves through a fluid path, collecting data through sensors. Specifically, it captures omnidirectional video data with a camera, acquires vibration data with an accelerometer, and collects acoustic data with a microphone. The input is instantaneous environmental changes. This data is immediately compressed; for example, video data is encoded in H.264 format, and vibration and acoustic data in FLAC format. The output is compressed packaged data.

[0177] Step 2:

[0178] The terminal transmits the compressed data it has collected to the base station. The data is sent in real time via the LTE communication module. The input includes the data compressed in the previous step. Before transmission, the terminal adds a timestamp and device identification information to each packet of data to optimize it for transmission to the ground. The output is a data stream that can be received by the base station.

[0179] Step 3:

[0180] The server receives data from the base station and first processes it for denoising and normalization. The server uses Python to remove high-frequency noise from the received data and the NumPy library to normalize the data to a standard scale. The input is compressed data transmitted from the base station. The output is clean, analyzable data.

[0181] Step 4:

[0182] The server uses a machine learning model to analyze denoised data. The model, built using the TensorFlow library, detects anomalies in the data and identifies potential problems. The input is normalized data. This process identifies the type and urgency of the anomalies and generates warnings about potential risks. The output consists of the detected anomalies and their associated analysis results.

[0183] Step 5:

[0184] The user views the analysis results through a user interface. The user interface displays detailed analysis results and provides information optimized by the sentiment analysis engine. Inputs are analysis results from the server and the user's real-time sentiment data. Specifically, it displays anomalies on a map and summarizes information according to its urgency. Outputs are visual reports and status notifications that the user can review.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Conventional infrastructure management systems, while capable of detecting and predicting anomalies in fluid pathways, lacked the ability to provide information tailored to the user's emotional state, resulting in insufficient improvements in work efficiency and stress reduction. Furthermore, when an anomaly actually occurs, users need real-time information and support to take appropriate action. However, existing systems have struggled to meet these needs.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes emotion recognition means for evaluating the user's emotional state and determining how to present information; means for notifying the user of information regarding the detection or prediction of anomalies visually and audibly; and means for a user interface that displays information on anomaly locations identified on a map, presents detailed analysis results, and provides stress reduction information tailored to the user's emotional state. This enables the user to access anomaly information in real time, reduce stress while responding efficiently, and improve work efficiency.

[0190] A "fluid pathway" is a path through which fluids such as liquids and gases move, and includes infrastructure structures such as pipelines and duct systems.

[0191] "Omnidirectional video data" refers to video data captured in all 360 degrees, recording the entire surroundings from a specific point.

[0192] "Vibration data" refers to data that shows the characteristics of vibrations obtained from fluid paths and the surrounding environment, and is information used to detect anomalies.

[0193] "Audio data" refers to data that shows acoustic signals collected from within or around a fluid path, and is useful for detecting anomalies and evaluating the environment.

[0194] A "learning model" is a computational method that uses machine learning algorithms to analyze data and identify patterns and anomalies.

[0195] "Emotion recognition means" refers to technology that analyzes a user's emotional state from their voice and facial expressions and generates appropriate feedback.

[0196] A "user interface" is a visual and operational interface for the exchange of information between the user and the system, and it presents analysis results and detailed information.

[0197] "Notification means" refers to visual or audio technologies used to inform users of information based on the detection or prediction of anomalies.

[0198] To implement this invention, it is necessary to construct a system for monitoring fluid paths and detecting anomalies. The terminal is integrated into a device such as smart glasses and collects omnidirectional video, vibration, and audio data while moving within the fluid path. This terminal transmits the collected data to a ground base station in real time. The hardware used would be smart glasses (e.g., HoloLens®) and high-sensitivity sensors.

[0199] The server performs noise reduction and normalization on the received data and analyzes it using a learning model for anomaly detection. Based on this analysis, it detects anomalies and, if necessary, predicts signs of anomalies. Furthermore, it uses emotion recognition means to evaluate the user's emotional state from their voice and facial expressions and optimizes the information presentation method. The software used is an analysis program equipped with machine learning algorithms and an emotion recognition engine.

[0200] Through the user interface of the smart glasses, users can visually identify anomalies on a map and obtain detailed analysis results. Notifications visually and audibly alert the user to the presence or absence of anomalies, and stress reduction information tailored to their emotional state is provided. For example, if a pressure anomaly is detected during work, the glasses will display "Appropriate action is required regarding the detected pressure anomaly," and if stress is detected from the user's facial expression, an audible message will say, "It is important to proceed calmly in the next steps."

[0201] An example of a prompt message might be something like, "Consider providing information that is effective in detecting factory anomalies and reducing user stress." This allows the system to support users' efficient work and improve the overall security of infrastructure management.

[0202] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0203] Step 1:

[0204] The device moves through a fluid path while collecting omnidirectional video, vibration, and audio data in real time. In this step, smart glasses and sensors are used to convert the collected data into a digital format. The input here is physical video, vibration, and audio information, and the output is data in digital format.

[0205] Step 2:

[0206] The terminal collects digital data and transmits it to a ground base station in real time. Secure and high-speed data transfer is performed via a communication module. Inputs are digital video, vibration, and audio data, and output is the data transmission status to the base station.

[0207] Step 3:

[0208] The server performs noise reduction and normalization on the data received from the base station. This involves cleaning and scaling the data and converting it into a format suitable for analysis. The input is the data received from the base station, and the output is the processed, clean data.

[0209] Step 4:

[0210] The server uses a machine learning model to detect anomalies in processed data. Machine learning algorithms are used to identify anomaly patterns and predict future occurrences. The input is processed, clean data, and the output includes information on the presence or absence of anomalies and predictions.

[0211] Step 5:

[0212] The server uses emotion recognition to evaluate the user's emotional state. It analyzes voice and facial expression data to determine the user's psychological state. The input is the user's voice and facial expression data, and the output is the evaluation result of the emotional state.

[0213] Step 6:

[0214] Users view anomaly information and analysis results through smart glasses. The user interface visually displays anomalies on a map and provides details. Input is analysis results from the server, and output is visual and audio feedback to the user.

[0215] Step 7:

[0216] The server considers the user's emotional state and provides stress-reducing information and guidance. Using a generative AI model, it optimizes information presentation methods and generates prompts tailored to the user's needs. The input is the result of the emotional state assessment, and the output is the optimized information presentation.

[0217] This allows users to grasp anomaly information in real time and perform tasks more effectively.

[0218] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0219] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0220] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0221] [Second Embodiment]

[0222] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0223] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0224] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0225] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0226] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0227] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0228] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0229] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0230] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0231] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0232] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0233] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0234] The system of the present invention consists of a specially designed device and its control software, as well as a server program and user interface for data analysis, in order to efficiently detect anomalies in a fluid path. As an embodiment of this invention, the specific processing details of the program are described below in natural language.

[0235] terminal

[0236] First, the terminal controls a device placed within the fluid path. This device is equipped with a camera that captures omnidirectional video and various sensors that collect vibration and audio data. The terminal controls the device to move at an appropriate speed within the fluid path and simultaneously issues commands to capture data at regular intervals. This allows the data to be packaged and transmitted to the base station in real time.

[0237] server

[0238] The server receives data sent from the ground base station and prepares it for data analysis. The server program integrates this data and uses a specific learning model to identify patterns of abnormal operation. For example, if the server detects an abnormal sound, it analyzes its frequency components and predicts the need for future repairs based on the patterns identified as abnormal, and notifies the user accordingly.

[0239] User

[0240] Users can review the analysis results in detail through the provided user interface. The interface plots anomalies on a map of the fluid path and displays detailed information about each anomaly, helping users respond quickly and accurately. Based on this information, users can decide whether to dispatch a maintenance team or take countermeasures, and contribute to improving the accuracy of the system by providing feedback.

[0241] This embodiment of the invention enables significantly more efficient and precise monitoring and maintenance of fluid pathways than conventional methods.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] The terminal activates the devices placed within the fluid path. Once the devices are activated, the omnidirectional cameras and various sensors begin operating and perform initial calibration. After calibration is complete, the devices begin moving along the configured path at a constant speed.

[0245] Step 2:

[0246] The terminal instructs the moving device to periodically collect omnidirectional video and sensor data. The device generates data in packages at specified time intervals, making them available for the terminal to receive. These packages contain video, vibration, and audio data along with timestamps.

[0247] Step 3:

[0248] The terminal transmits the collected data to a ground base station in real time. The data is transmitted wirelessly and received by a server. The terminal also adds location information to each data package, allowing the collection point of each data to be identified.

[0249] Step 4:

[0250] The server receives data packages transmitted from ground base stations, verifies their integrity, and then prepares them for analysis. First, the server filters and denoises the data to improve processing efficiency, and then converts it into a format suitable for the learning model.

[0251] Step 5:

[0252] The server inputs the adjusted data into a learning model and executes the anomaly detection process. Specifically, it detects anomaly patterns and identifies predicted risks. Once the analysis results are obtained, they are compiled and saved as a report.

[0253] Step 6:

[0254] Users access analysis reports generated from the server through a user interface. The interface displays each anomaly on a map and presents detailed information about the detected locations. This allows users to take immediate corrective action.

[0255] Step 7:

[0256] Users decide on actions to take to address any anomalies they discover, such as dispatching a maintenance team as needed. Users can also feed back field-tested data to the server, which contributes to improving the overall accuracy of the system.

[0257] (Example 1)

[0258] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0259] Early detection of anomalies in fluid pathways is crucial, but current technologies often delay detection, potentially leading to serious problems. Furthermore, when anomalies are detected, accurate location information and detailed analysis results are necessary, requiring efficient and sophisticated technology. Therefore, there is a need for the development of new systems that enable rapid and accurate anomaly detection, prediction, and information provision.

[0260] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0261] In this invention, the server includes means for integrating and analyzing omnidirectional video information, vibration information, and audio information within a fluid path; means for detecting anomalies and predicting signs of anomaly occurrence based on the analysis results; means for transmitting analysis information in real time via a communication network; and means for providing a human-machine interface for visualizing the analyzed information and identifying the location of anomalies. This enables rapid and accurate anomaly detection and prediction, as well as effective response through the provision of more detailed information.

[0262] A "fluid pathway" refers to a passage through which a liquid or gas moves, and includes pipelines and ducts.

[0263] "Omnidirectional video information" refers to video data that can capture the entire 360-degree field of view, encompassing all visual information of the surroundings from a specific location.

[0264] "Vibration information" refers to data on the physical vibrations of objects and structures, and is information on periodic motion measured by sensors.

[0265] "Audio information" refers to data that describes the characteristics and properties of sound, and is obtained through the capture of sound signals by sensors.

[0266] An "information analysis device" refers to a device that receives various types of information as input, integrates and analyzes them, and outputs the results.

[0267] A "learning algorithm" refers to a method and procedure for learning patterns based on input data and performing analysis or prediction.

[0268] A "human-machine interface" is a means and interface for a user to interact with a machine, and includes elements that enable the visualization and input of information.

[0269] A "communication network" refers to a system for transmitting and sharing information, and includes means for connecting computers and other devices.

[0270] "Anomaly detection" refers to the process or ability to identify a state that deviates from normal operation.

[0271] "Predictive forecasting" refers to analysis and prediction aimed at foreseeing potential future changes or anomalies.

[0272] The system of this invention is designed to efficiently detect anomalies in fluid pathways. Specific embodiments of each component are shown below.

[0273] terminal

[0274] The terminal controls sensor-equipped devices placed within the fluid path. These devices are equipped with cameras for capturing omnidirectional video information and various sensors for acquiring vibration and audio information. The terminal captures and packages data in real time as the devices move within the fluid path. The collected data is transmitted to a ground base station via a communication network. Typical communication technologies used include Wi-Fi and mobile data communication.

[0275] server

[0276] The server is a device that receives and analyzes data transmitted through ground base stations. The server is equipped with a learning algorithm that identifies anomaly patterns based on the integrated data. This allows the server to perform anomaly detection in real time. For example, generative AI models are used to recognize abnormal vibrations and sound patterns during the analysis. If an anomaly is detected, the server predicts its precursors and notifies the user of the necessary information.

[0277] User

[0278] Users can review the system's analysis results and obtain detailed information about anomalies. Through a dedicated human-machine interface, users can access detailed information, including data visualizations, enabling rapid response. The interface allows users to issue maintenance instructions and arrange for repairs to anomalies. Furthermore, the feedback received can be used to improve the system.

[0279] For example, if an abnormal sound pattern is detected within the fluid path, the location will be displayed in red on the map, and detailed frequency analysis data will be provided. Furthermore, information will be provided to the user through prompt messages such as, "Analyze the sound and vibration data collected by devices installed within the fluid path, and notify us if any abnormalities are found. Display the location of the abnormality on the map so that maintenance can be arranged as needed."

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The terminal activates sensor-equipped devices placed within the fluid path to collect omnidirectional video, vibration, and audio information. The input here is raw data from each sensor, which the terminal converts into a digital format and packages. This data package is transmitted to a ground base station via Wi-Fi or a mobile data network. Specifically, the device captures data every second as it moves and transmits it at regular time intervals.

[0283] Step 2:

[0284] The server decompresses the data packages received from the terrestrial base station and checks the data integrity. The input is the packaged data transmitted from the terminal, and the output is the integrated data in a format suitable for analysis. The server sorts this based on the time axis and prepares for anomaly detection. At this time, if there is missing data, an operation for predictive completion is inserted.

[0285] Step 3:

[0286] The server performs analysis using the integrated data to utilize the generated AI model. The input here is the prepared data obtained in Step 2, and the output is the anomaly detection and prediction results. The server uses a learning algorithm to extract the features of the data and identify abnormal patterns. If an anomaly is recognized, a prediction of future anomaly occurrence is made based on its features. For example, an operation of analyzing the frequency components of voice data and comparing with past abnormal cases is performed.

[0287] Step 4:

[0288] The server converts the analysis results into a format understandable to the user and presents them through a dedicated human-machine interface. The input is the analysis results obtained in Step 3, and the output is the visualized information and specific recommendations. For example, an operation of plotting abnormal locations on a map of the fluid path and displaying the specific causes and countermeasure plans for the anomalies is performed.

[0289] Step 5:

[0290] The user makes a quick and accurate decision based on the information obtained through the interface. The input is the analysis information provided by the server, and the user evaluates this and issues a maintenance instruction. By providing feedback, it can also contribute to further improvement of the system. Specific operations include clicking on the abnormal location to check detailed information and sending an instruction to the maintenance team by email.

[0291] (Application Example 1)

[0292] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0293] It is crucial to detect abnormalities in fluid pathways early and prevent unexpected failures of machinery and equipment. However, predicting abnormalities is currently difficult, requiring rapid and precise monitoring and judgment. Furthermore, there is a lack of visual information to smoothly identify abnormal locations and instruct corrective actions.

[0294] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0295] In this invention, the server includes information analysis means that include a model for integrating and analyzing omnidirectional visual data, vibration data, and acoustic data; means for detecting abnormalities and predicting signs of anomaly occurrence; means for transmitting sequential information via wireless communication; and means for visualizing the results of anomaly detection based on time and location information. This enables early detection of anomalies in the fluid path, accurate location identification, and rapid implementation of countermeasures.

[0296] A "device that moves within a fluid passage" is a device that moves within a fluid passage while acquiring information about its surroundings.

[0297] "Omnidirectional visual data" refers to data that includes video information of the entire surrounding area at a given point.

[0298] "Vibration data" refers to data that includes information about vibrations within a fluid passage.

[0299] "Acoustic data" refers to data that contains information related to sounds generated within a fluid passage.

[0300] An "information analysis device" is a device designed to analyze collected data.

[0301] A "model" is a calculation method or algorithm used to identify specific patterns.

[0302] "Abnormal" is a term referring to an abnormal state that deviates from the normal range.

[0303] "Sequential information" is a term referring to data that is transmitted immediately upon being collected.

[0304] "Wireless communication" is a method of information transmission to a remote location using radio waves.

[0305] "Visualization" refers to presenting data and information in a form that is easy for people to understand.

[0306] "Time and location information" is data that includes information about a geographical location at a specific point in time.

[0307] The system for implementing this invention provides a comprehensive function for monitoring fluid paths in a factory. Each of its elements and operation processes will be described below.

[0308] The terminal moves inside the fluid path and acquires data through visual sensors and acoustic sensors installed in multiple directions. These sensors are used to collect omnidirectional visual data, vibration data, and acoustic data. This data is continuously transmitted from the terminal to the server via wireless communication.

[0309] The server is a central device that receives the collected data and performs information analysis. The server has an information analysis device built in a Python environment and is equipped with an advanced calculation model using TensorFlow. This model executes an algorithm that identifies abnormal patterns and predicts the possibility of abnormal occurrences at an early stage. Furthermore, when an abnormality is detected, the location is specified based on the data, and visualization processing is performed based on the time and location information.

[0310] Users can view detailed analysis results through an administration screen designed with React.js. This user interface clearly displays the location of detected anomalies on a map, helping users quickly identify problems and take appropriate action.

[0311] As a concrete example, when introducing new machinery to a factory, this system can be used to monitor the fluid pathways, allowing for the rapid detection of problems that tend to occur in the initial stages of implementation, thereby preventing major failures. By utilizing a generative AI model, for example, by inputting a prompt such as, "If abnormal vibrations or sounds are detected in the fluid pathways of newly introduced machinery, please list their locations and countermeasures," it is possible to compare them with known abnormality patterns and suggest appropriate countermeasures.

[0312] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0313] Step 1:

[0314] The terminal collects omnidirectional visual data, vibration data, and acoustic data from multidirectional sensors installed within the fluid channel. The input is real-world environmental information within the fluid channel, and the output is digital data that packages this data together. This data is acquired directly from the sensor devices and processed in real time.

[0315] Step 2:

[0316] The terminal transmits the collected digital data to the server via wireless communication. The input is the digital data packaged in step 1, and the output is the data received by the server. Specifically, the Wi-Fi module manages the data transmission and applies the appropriate protocol to prevent data loss.

[0317] Step 3:

[0318] The server feeds the received digital data into an analysis model to identify anomalous patterns. The input is the digital data received by the server, and the output is the analysis results of the patterns and signs of anomalies that have been determined to be anomalous. TensorFlow is used for this analysis, and the specific computational processes for anomaly detection are performed using machine learning algorithms.

[0319] Step 4:

[0320] The server identifies the location of the anomaly based on the analysis results, adds time and location information, and generates visualization data. The input is the analysis results obtained in step 3, and the output is the visualized anomaly information. Specifically, it integrates map information and analysis results, and highlights the necessary information using a visualization algorithm.

[0321] Step 5:

[0322] Users review visualized anomaly information through the management screen and determine the necessary countermeasures. The input is the visualized data provided in step 4, and the output is the countermeasures and action plan decided by the user. Specifically, the user interface effectively indicates the location of the anomaly and assists the user in making quick decisions.

[0323] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0324] The present invention provides an advanced infrastructure management system that, in addition to monitoring fluid pathways and detecting anomalies, also has the ability to recognize the emotional state of the user. An embodiment of the system is configured as follows:

[0325] terminal

[0326] The terminal controls a device installed within the fluid path. This device has a set of sensors to capture omnidirectional video and collect vibration and audio data. The device moves along the fluid path, packaging the data collected by the terminal and transmitting it in real time to a ground base station.

[0327] server

[0328] The server processes data received from the base station. The received dataset undergoes predetermined noise reduction and data normalization before being analyzed by a learning model. Based on the analysis results, the server quickly detects anomalies, predicts potential risks, and takes appropriate action. Furthermore, this server program incorporates an emotion engine to evaluate user reactions based on the analysis results.

[0329] User

[0330] Users can view anomaly detection reports in real time through a user interface that displays data analysis information. The user interface visually shows detected anomalies on a map and provides detailed reports. Furthermore, the emotion engine analyzes the user's voice and facial expressions to automatically select the optimal information presentation method. This plays a role in increasing efficiency in the user's daily work and supporting accurate maintenance tasks.

[0331] As a concrete example, when users encounter complex anomaly analysis results, the emotion engine summarizes and presents the information to reduce their stress levels. For instance, if the urgency of an anomaly is judged to be low, the notification can be presented in simple language to avoid confusing the user.

[0332] This embodiment enables the system to adapt to environmental changes, improve the user experience, and achieve secure and efficient infrastructure management.

[0333] The following describes the processing flow.

[0334] Step 1:

[0335] The terminal activates the device placed within the fluid path and prepares to capture omnidirectional video. After activation, the device uses its omnidirectional camera and various sensors to collect video, vibration, and audio data as it moves along the designated route.

[0336] Step 2:

[0337] The terminal packages the collected data in real time and then transmits it to a ground base station. The package includes all the collected data as well as location information, which allows the data collection point to be identified.

[0338] Step 3:

[0339] The server receives data from the ground base station and preprocesses it by performing noise reduction and data filtering. The preprocessed data is then fed into a learning model to perform anomaly detection and risk prediction.

[0340] Step 4:

[0341] The server generates an anomaly report based on the analysis results. The report includes the type and location of the detected anomaly, as well as risk information that requires immediate action if necessary. Furthermore, the server uses an emotion engine to analyze the user's current emotional data in order to recognize the user's emotional state.

[0342] Step 5:

[0343] After the emotion engine evaluates the user's emotional state, the server uses the evaluation results to notify the user of an anomaly in the most appropriate way. For example, if the user is experiencing stress, the server will respond by presenting the information in a simplified format.

[0344] Step 6:

[0345] Users view analysis reports from the server through a user interface. The interface displays anomalies on a map, and detailed information for each anomaly is provided in an easily accessible format.

[0346] Step 7:

[0347] Users plan appropriate actions based on the report. Specifically, they may dispatch a maintenance team or arrange for additional inspection work. They can also contribute to continuous improvement by providing feedback to the system.

[0348] (Example 2)

[0349] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0350] In modern infrastructure, early detection of anomalies and rapid and effective information provision to users are crucial. However, conventional systems lacked sufficient accuracy in data noise reduction, assessment of anomaly urgency, and appropriate information presentation methods tailored to users' emotional states. As a result, on-site responses were delayed, leading to increased user stress. This invention aims to solve these problems and support safe and efficient infrastructure operation.

[0351] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0352] In this invention, the server includes means for denoising and normalizing data, means for analyzing anomalies using a learning model and evaluating their urgency, and means for evaluating the user's emotional state using sentiment analysis technology and providing optimal information. This enables real-time anomaly detection and the provision of appropriate information to reduce user stress.

[0353] A "fluid pathway" is a passage or pipe through which a fluid, such as a liquid or gas, moves.

[0354] "Video data" refers to digital data containing visual information acquired using devices such as cameras.

[0355] "Vibration data" refers to information about vibrations of objects or the environment that are recorded using sensors.

[0356] "Acoustic data" refers to data collected by sound sensors that includes information about the surrounding sounds.

[0357] "Data processing means" refers to the system's functions, including the process of removing noise and normalizing acquired data.

[0358] A "learning model" is a model that analyzes data based on machine learning algorithms to perform pattern recognition and anomaly detection.

[0359] "Anomaly detection" is the process of identifying unusual data patterns or behaviors that indicate potential problems in a system.

[0360] "Emotional analysis technology" is a technology that analyzes voice and facial expression data to evaluate a user's emotional state.

[0361] A "user interface" refers to the screens and devices that a user uses to interact with a system and obtain information.

[0362] "Optimal information presentation" means providing information in a format that is easy to understand and reduces stress, tailored to the user's current emotional state.

[0363] The present invention is an advanced infrastructure management technology aimed at early detection of anomalies in fluid pathways and providing users with appropriate information. This system collects data using a group of devices installed in the fluid pathway, analyzes it using a server, and provides an interface for users to view the information.

[0364] terminal

[0365] The terminal moves along a fluid path and controls multiple sensors. Specifically, it is equipped with a camera that captures omnidirectional video, an accelerometer that detects vibrations, and a microphone that picks up sound. The terminal compresses and packages the data obtained from these sensors in real time and transmits it to the base station. A dedicated communication module is used for data transmission, enabling real-time communication.

[0366] server

[0367] The server receives data transmitted from the base station and first performs noise reduction. It cleanses and normalizes the data using programming libraries such as Python and NumPy. Next, it runs an analysis model on the processed data using machine learning libraries such as TensorFlow to detect anomalies and assess their urgency. Furthermore, it integrates the analysis results with sentiment analysis technology to evaluate the user's emotional state and provide appropriate information.

[0368] User

[0369] Users can view these analysis results through a user interface. This interface operates on a web browser and is designed to allow users to intuitively understand the anomalies and their details. Based on sentiment analysis technology, the format of information presentation is adjusted according to the user's current emotional state, playing a role in reducing stress and confusion.

[0370] As a concrete example, consider a case where abnormal vibrations are detected in a part of the fluid path. Based on this data, the server quickly identifies the anomaly and provides the user with a notification such as, "Abnormal vibration detected: Vibration confirmed in fluid path sector 3. No immediate action is required," thereby supporting the user in responding calmly.

[0371] The following are some possible prompt statements for a generative AI model.

[0372] "Summarize the latest anomaly analysis results in fluid pathways and propose an appropriate information presentation method to reduce user stress."

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The device moves through a fluid path, collecting data through sensors. Specifically, it captures omnidirectional video data with a camera, acquires vibration data with an accelerometer, and collects acoustic data with a microphone. The input is instantaneous environmental changes. This data is immediately compressed; for example, video data is encoded in H.264 format, and vibration and acoustic data in FLAC format. The output is compressed packaged data.

[0376] Step 2:

[0377] The terminal transmits the compressed data it has collected to the base station. The data is sent in real time via the LTE communication module. The input includes the data compressed in the previous step. Before transmission, the terminal adds a timestamp and device identification information to each packet of data to optimize it for transmission to the ground. The output is a data stream that can be received by the base station.

[0378] Step 3:

[0379] The server receives data from the base station and first processes it for denoising and normalization. The server uses Python to remove high-frequency noise from the received data and the NumPy library to normalize the data to a standard scale. The input is compressed data transmitted from the base station. The output is clean, analyzable data.

[0380] Step 4:

[0381] The server uses a machine learning model to analyze denoised data. The model, built using the TensorFlow library, detects anomalies in the data and identifies potential problems. The input is normalized data. This process identifies the type and urgency of the anomalies and generates warnings about potential risks. The output consists of the detected anomalies and their associated analysis results.

[0382] Step 5:

[0383] The user views the analysis results through a user interface. The user interface displays detailed analysis results and provides information optimized by the sentiment analysis engine. Inputs are analysis results from the server and the user's real-time sentiment data. Specifically, it displays anomalies on a map and summarizes information according to its urgency. Outputs are visual reports and status notifications that the user can review.

[0384] (Application Example 2)

[0385] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0386] Conventional infrastructure management systems, while capable of detecting and predicting anomalies in fluid pathways, lacked the ability to provide information tailored to the user's emotional state, resulting in insufficient improvements in work efficiency and stress reduction. Furthermore, when an anomaly actually occurs, users need real-time information and support to take appropriate action. However, existing systems have struggled to meet these needs.

[0387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0388] In this invention, the server includes emotion recognition means for evaluating the user's emotional state and determining how to present information; means for notifying the user of information regarding the detection or prediction of anomalies visually and audibly; and means for a user interface that displays information on anomaly locations identified on a map, presents detailed analysis results, and provides stress reduction information tailored to the user's emotional state. This enables the user to access anomaly information in real time, reduce stress while responding efficiently, and improve work efficiency.

[0389] A "fluid pathway" is a path through which fluids such as liquids and gases move, and includes infrastructure structures such as pipelines and duct systems.

[0390] "Omnidirectional video data" refers to video data captured in all 360 degrees, recording the entire surroundings from a specific point.

[0391] "Vibration data" refers to data that shows the characteristics of vibrations obtained from fluid paths and the surrounding environment, and is information used to detect anomalies.

[0392] "Audio data" refers to data that shows acoustic signals collected from within or around a fluid path, and is useful for detecting anomalies and evaluating the environment.

[0393] A "learning model" is a computational method that uses machine learning algorithms to analyze data and identify patterns and anomalies.

[0394] "Emotion recognition means" refers to technology that analyzes a user's emotional state from their voice and facial expressions and generates appropriate feedback.

[0395] A "user interface" is a visual and operational interface for the exchange of information between the user and the system, and it presents analysis results and detailed information.

[0396] "Notification means" refers to visual or audio technologies used to inform users of information based on the detection or prediction of anomalies.

[0397] To implement this invention, it is necessary to construct a system for monitoring fluid paths and detecting anomalies. The terminal is integrated into a device such as smart glasses and collects omnidirectional video, vibration, and audio data while moving within the fluid path. This terminal transmits the collected data to a ground base station in real time. The hardware used would be smart glasses (e.g., HoloLens) and high-sensitivity sensors.

[0398] The server performs noise reduction and normalization on the received data and analyzes it using a learning model for anomaly detection. Based on this analysis, it detects anomalies and, if necessary, predicts signs of anomalies. Furthermore, it uses emotion recognition means to evaluate the user's emotional state from their voice and facial expressions and optimizes the information presentation method. The software used is an analysis program equipped with machine learning algorithms and an emotion recognition engine.

[0399] Through the user interface of the smart glasses, users can visually identify anomalies on a map and obtain detailed analysis results. Notifications visually and audibly alert the user to the presence or absence of anomalies, and stress reduction information tailored to their emotional state is provided. For example, if a pressure anomaly is detected during work, the glasses will display "Appropriate action is required regarding the detected pressure anomaly," and if stress is detected from the user's facial expression, an audible message will say, "It is important to proceed calmly in the next steps."

[0400] An example of a prompt message might be something like, "Consider providing information that is effective in detecting factory anomalies and reducing user stress." This allows the system to support users' efficient work and improve the overall security of infrastructure management.

[0401] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0402] Step 1:

[0403] The device moves through a fluid path while collecting omnidirectional video, vibration, and audio data in real time. In this step, smart glasses and sensors are used to convert the collected data into a digital format. The input here is physical video, vibration, and audio information, and the output is data in digital format.

[0404] Step 2:

[0405] The terminal collects digital data and transmits it to a ground base station in real time. Secure and high-speed data transfer is performed via a communication module. Inputs are digital video, vibration, and audio data, and output is the data transmission status to the base station.

[0406] Step 3:

[0407] The server performs noise reduction and normalization on the data received from the base station. This involves cleaning and scaling the data and converting it into a format suitable for analysis. The input is the data received from the base station, and the output is the processed, clean data.

[0408] Step 4:

[0409] The server uses a machine learning model to detect anomalies in processed data. Machine learning algorithms are used to identify anomaly patterns and predict future occurrences. The input is processed, clean data, and the output includes information on the presence or absence of anomalies and predictions.

[0410] Step 5:

[0411] The server uses emotion recognition to evaluate the user's emotional state. It analyzes voice and facial expression data to determine the user's psychological state. The input is the user's voice and facial expression data, and the output is the evaluation result of the emotional state.

[0412] Step 6:

[0413] Users view anomaly information and analysis results through smart glasses. The user interface visually displays anomalies on a map and provides details. Input is analysis results from the server, and output is visual and audio feedback to the user.

[0414] Step 7:

[0415] The server considers the user's emotional state and provides stress-reducing information and guidance. Using a generative AI model, it optimizes information presentation methods and generates prompts tailored to the user's needs. The input is the result of the emotional state assessment, and the output is the optimized information presentation.

[0416] This allows users to grasp anomaly information in real time and perform tasks more effectively.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0418] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0420] [Third Embodiment]

[0421] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0422] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0423] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0425] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0427] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0428] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0429] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0432] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0433] The system of the present invention consists of a specially designed device and its control software, as well as a server program and user interface for data analysis, in order to efficiently detect anomalies in a fluid path. As an embodiment of this invention, the specific processing details of the program are described below in natural language.

[0434] terminal

[0435] First, the terminal controls a device placed within the fluid path. This device is equipped with a camera that captures omnidirectional video and various sensors that collect vibration and audio data. The terminal controls the device to move at an appropriate speed within the fluid path and simultaneously issues commands to capture data at regular intervals. This allows the data to be packaged and transmitted to the base station in real time.

[0436] server

[0437] The server receives data sent from the ground base station and prepares it for data analysis. The server program integrates this data and uses a specific learning model to identify patterns of abnormal operation. For example, if the server detects an abnormal sound, it analyzes its frequency components and predicts the need for future repairs based on the patterns identified as abnormal, and notifies the user accordingly.

[0438] User

[0439] Users can review the analysis results in detail through the provided user interface. The interface plots anomalies on a map of the fluid path and displays detailed information about each anomaly, helping users respond quickly and accurately. Based on this information, users can decide whether to dispatch a maintenance team or take countermeasures, and contribute to improving the accuracy of the system by providing feedback.

[0440] This embodiment of the invention enables significantly more efficient and precise monitoring and maintenance of fluid pathways than conventional methods.

[0441] The following describes the processing flow.

[0442] Step 1:

[0443] The terminal activates the devices placed within the fluid path. Once the devices are activated, the omnidirectional cameras and various sensors begin operating and perform initial calibration. After calibration is complete, the devices begin moving along the configured path at a constant speed.

[0444] Step 2:

[0445] The terminal instructs the moving device to periodically collect omnidirectional video and sensor data. The device generates data in packages at specified time intervals, making them available for the terminal to receive. These packages contain video, vibration, and audio data along with timestamps.

[0446] Step 3:

[0447] The terminal transmits the collected data to a ground base station in real time. The data is transmitted wirelessly and received by a server. The terminal also adds location information to each data package, allowing the collection point of each data to be identified.

[0448] Step 4:

[0449] The server receives data packages transmitted from ground base stations, verifies their integrity, and then prepares them for analysis. First, the server filters and denoises the data to improve processing efficiency, and then converts it into a format suitable for the learning model.

[0450] Step 5:

[0451] The server inputs the adjusted data into a learning model and executes the anomaly detection process. Specifically, it detects anomaly patterns and identifies predicted risks. Once the analysis results are obtained, they are compiled and saved as a report.

[0452] Step 6:

[0453] Users access analysis reports generated from the server through a user interface. The interface displays each anomaly on a map and presents detailed information about the detected locations. This allows users to take immediate corrective action.

[0454] Step 7:

[0455] Users decide on actions to take to address any anomalies they discover, such as dispatching a maintenance team as needed. Users can also feed back field-tested data to the server, which contributes to improving the overall accuracy of the system.

[0456] (Example 1)

[0457] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0458] Early detection of anomalies in fluid pathways is crucial, but current technologies often delay detection, potentially leading to serious problems. Furthermore, when anomalies are detected, accurate location information and detailed analysis results are necessary, requiring efficient and sophisticated technology. Therefore, there is a need for the development of new systems that enable rapid and accurate anomaly detection, prediction, and information provision.

[0459] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0460] In this invention, the server includes means for integrating and analyzing omnidirectional video information, vibration information, and audio information within a fluid path; means for detecting anomalies and predicting signs of anomaly occurrence based on the analysis results; means for transmitting analysis information in real time via a communication network; and means for providing a human-machine interface for visualizing the analyzed information and identifying the location of anomalies. This enables rapid and accurate anomaly detection and prediction, as well as effective response through the provision of more detailed information.

[0461] A "fluid pathway" refers to a passage through which a liquid or gas moves, and includes pipelines and ducts.

[0462] "Omnidirectional video information" refers to video data that can capture the entire 360-degree field of view, encompassing all visual information of the surroundings from a specific location.

[0463] "Vibration information" refers to data on the physical vibrations of objects and structures, and is information on periodic motion measured by sensors.

[0464] "Audio information" refers to data that describes the characteristics and properties of sound, and is obtained through the capture of sound signals by sensors.

[0465] An "information analysis device" refers to a device that receives various types of information as input, integrates and analyzes them, and outputs the results.

[0466] A "learning algorithm" refers to a method and procedure for learning patterns based on input data and performing analysis or prediction.

[0467] A "human-machine interface" is a means and interface for a user to interact with a machine, and includes elements that enable the visualization and input of information.

[0468] A "communication network" refers to a system for transmitting and sharing information, and includes means for connecting computers and other devices.

[0469] "Anomaly detection" refers to the process or ability to identify a state that deviates from normal operation.

[0470] "Predictive forecasting" refers to analysis and prediction aimed at foreseeing potential future changes or anomalies.

[0471] The system of this invention is designed to efficiently detect anomalies in fluid pathways. Specific embodiments of each component are shown below.

[0472] terminal

[0473] The terminal controls sensor-equipped devices placed within the fluid path. These devices are equipped with cameras for capturing omnidirectional video information and various sensors for acquiring vibration and audio information. The terminal captures and packages data in real time as the devices move within the fluid path. The collected data is transmitted to a ground base station via a communication network. Typical communication technologies used include Wi-Fi and mobile data communication.

[0474] server

[0475] The server is a device that receives and analyzes data transmitted through ground base stations. The server is equipped with a learning algorithm that identifies anomaly patterns based on the integrated data. This allows the server to perform anomaly detection in real time. For example, generative AI models are used to recognize abnormal vibrations and sound patterns during the analysis. If an anomaly is detected, the server predicts its precursors and notifies the user of the necessary information.

[0476] User

[0477] Users can review the system's analysis results and obtain detailed information about anomalies. Through a dedicated human-machine interface, users can access detailed information, including data visualizations, enabling rapid response. The interface allows users to issue maintenance instructions and arrange for repairs to anomalies. Furthermore, the feedback received can be used to improve the system.

[0478] For example, if an abnormal sound pattern is detected within the fluid path, the location will be displayed in red on the map, and detailed frequency analysis data will be provided. Furthermore, information will be provided to the user through prompt messages such as, "Analyze the sound and vibration data collected by devices installed within the fluid path, and notify us if any abnormalities are found. Display the location of the abnormality on the map so that maintenance can be arranged as needed."

[0479] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0480] Step 1:

[0481] The terminal activates sensor-equipped devices placed within the fluid path to collect omnidirectional video, vibration, and audio information. The input here is raw data from each sensor, which the terminal converts into a digital format and packages. This data package is transmitted to a ground base station via Wi-Fi or a mobile data network. Specifically, the device captures data every second as it moves and transmits it at regular time intervals.

[0482] Step 2:

[0483] The server decompresses the data packages received from the ground base station and verifies the data's integrity. The input is packaged data transmitted from the terminal, and the output is integrated data in a format suitable for analysis. The server sorts this data based on a time axis and prepares it for anomaly detection. During this process, it inserts an operation to perform predictive imputation if there is missing data.

[0484] Step 3:

[0485] The server performs analysis using a generative AI model with the integrated data. The input here is the prepared data obtained in step 2, and the output is the anomaly detection and prediction results. The server uses a learning algorithm to extract features from the data and identify anomaly patterns. If an anomaly is recognized, it predicts future anomaly occurrences based on those features. For example, it analyzes the frequency components of audio data and compares them with past anomaly cases.

[0486] Step 4:

[0487] The server converts the analysis results into a format understandable to the user and presents them through a dedicated human-machine interface. The input is the analysis results obtained in step 3, and the output is visualized information and specific recommendations. For example, it can plot anomalies on a fluid path map and display the specific causes of the anomalies and suggested countermeasures.

[0488] Step 5:

[0489] Users make quick and accurate decisions based on information obtained through the interface. The input is analytical information provided by the server, which users evaluate and issue maintenance instructions. Providing feedback also contributes to further system improvements. Specific actions include clicking on anomalies to view detailed information and sending instructions to the maintenance team via email.

[0490] (Application Example 1)

[0491] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0492] It is crucial to detect abnormalities in fluid pathways early and prevent unexpected failures of machinery and equipment. However, predicting abnormalities is currently difficult, requiring rapid and precise monitoring and judgment. Furthermore, there is a lack of visual information to smoothly identify abnormal locations and instruct corrective actions.

[0493] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0494] In this invention, the server includes information analysis means that include a model for integrating and analyzing omnidirectional visual data, vibration data, and acoustic data; means for detecting abnormalities and predicting signs of anomaly occurrence; means for transmitting sequential information via wireless communication; and means for visualizing the results of anomaly detection based on time and location information. This enables early detection of anomalies in the fluid path, accurate location identification, and rapid implementation of countermeasures.

[0495] A "device that moves within a fluid passage" is a device that moves within a fluid passage while acquiring information about its surroundings.

[0496] "Omnidirectional visual data" refers to data that includes video information of the entire surrounding area at a given point.

[0497] "Vibration data" refers to data that includes information about vibrations within a fluid passage.

[0498] "Acoustic data" refers to data that contains information related to sounds generated within a fluid passage.

[0499] An "information analysis device" is a device designed to analyze collected data.

[0500] A "model" is a computational method or algorithm used to identify a specific pattern.

[0501] "Abnormal" is a term that refers to an abnormal state that deviates from the normal range.

[0502] "Sequential information" is a term that refers to data that is collected and transmitted immediately.

[0503] "Wireless communication" is a method of transmitting information over long distances using radio waves.

[0504] "Visualization" refers to displaying data and information in a way that is easy for people to understand.

[0505] "Time and location information" refers to data that includes information about a geographical location at a specific point in time.

[0506] The system implementing this invention provides a comprehensive function for monitoring fluid pathways within a factory. Its elements and operating processes are described below.

[0507] The terminal moves within the fluid path and acquires data through multi-directionally positioned visual and acoustic sensors. These sensors are used to collect omnidirectional visual data, vibration data, and acoustic data. This data is continuously transmitted from the terminal to the server via wireless communication.

[0508] The server is the central device that receives the collected data and performs information analysis. The server has an information analysis system built in a Python environment and is equipped with an advanced computational model using TensorFlow. This model identifies anomaly patterns and executes algorithms to predict the likelihood of anomaly occurrences early on. Furthermore, if an anomaly is detected, it identifies its location based on the data and performs visualization processing based on time and location information.

[0509] Users can view detailed analysis results through an administration screen designed with React.js. This user interface clearly displays the location of detected anomalies on a map, helping users quickly identify problems and take appropriate action.

[0510] As a concrete example, when introducing new machinery to a factory, this system can be used to monitor the fluid pathways, allowing for the rapid detection of problems that tend to occur in the initial stages of implementation, thereby preventing major failures. By utilizing a generative AI model, for example, by inputting a prompt such as, "If abnormal vibrations or sounds are detected in the fluid pathways of newly introduced machinery, please list their locations and countermeasures," it is possible to compare them with known abnormality patterns and suggest appropriate countermeasures.

[0511] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0512] Step 1:

[0513] The terminal collects omnidirectional visual data, vibration data, and acoustic data from multidirectional sensors installed within the fluid channel. The input is real-world environmental information within the fluid channel, and the output is digital data that packages this data together. This data is acquired directly from the sensor devices and processed in real time.

[0514] Step 2:

[0515] The terminal transmits the collected digital data to the server via wireless communication. The input is the digital data packaged in step 1, and the output is the data received by the server. Specifically, the Wi-Fi module manages the data transmission and applies the appropriate protocol to prevent data loss.

[0516] Step 3:

[0517] The server feeds the received digital data into an analysis model to identify anomalous patterns. The input is the digital data received by the server, and the output is the analysis results of the patterns and signs of anomalies that have been determined to be anomalous. TensorFlow is used for this analysis, and the specific computational processes for anomaly detection are performed using machine learning algorithms.

[0518] Step 4:

[0519] The server identifies the location of the anomaly based on the analysis results, adds time and location information, and generates visualization data. The input is the analysis results obtained in step 3, and the output is the visualized anomaly information. Specifically, it integrates map information and analysis results, and highlights the necessary information using a visualization algorithm.

[0520] Step 5:

[0521] Users review visualized anomaly information through the management screen and determine the necessary countermeasures. The input is the visualized data provided in step 4, and the output is the countermeasures and action plan decided by the user. Specifically, the user interface effectively indicates the location of the anomaly and assists the user in making quick decisions.

[0522] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0523] The present invention provides an advanced infrastructure management system that, in addition to monitoring fluid pathways and detecting anomalies, also has the ability to recognize the emotional state of the user. An embodiment of the system is configured as follows:

[0524] terminal

[0525] The terminal controls a device installed within the fluid path. This device has a set of sensors to capture omnidirectional video and collect vibration and audio data. The device moves along the fluid path, packaging the data collected by the terminal and transmitting it in real time to a ground base station.

[0526] server

[0527] The server processes data received from the base station. The received dataset undergoes predetermined noise reduction and data normalization before being analyzed by a learning model. Based on the analysis results, the server quickly detects anomalies, predicts potential risks, and takes appropriate action. Furthermore, this server program incorporates an emotion engine to evaluate user reactions based on the analysis results.

[0528] User

[0529] Users can view anomaly detection reports in real time through a user interface that displays data analysis information. The user interface visually shows detected anomalies on a map and provides detailed reports. Furthermore, the emotion engine analyzes the user's voice and facial expressions to automatically select the optimal information presentation method. This plays a role in increasing efficiency in the user's daily work and supporting accurate maintenance tasks.

[0530] As a concrete example, when users encounter complex anomaly analysis results, the emotion engine summarizes and presents the information to reduce their stress levels. For instance, if the urgency of an anomaly is judged to be low, the notification can be presented in simple language to avoid confusing the user.

[0531] This embodiment enables the system to adapt to environmental changes, improve the user experience, and achieve secure and efficient infrastructure management.

[0532] The following describes the processing flow.

[0533] Step 1:

[0534] The terminal activates the device placed within the fluid path and prepares to capture omnidirectional video. After activation, the device uses its omnidirectional camera and various sensors to collect video, vibration, and audio data as it moves along the designated route.

[0535] Step 2:

[0536] The terminal packages the collected data in real time and then transmits it to a ground base station. The package includes all the collected data as well as location information, which allows the data collection point to be identified.

[0537] Step 3:

[0538] The server receives data from the ground base station and preprocesses it by performing noise reduction and data filtering. The preprocessed data is then fed into a learning model to perform anomaly detection and risk prediction.

[0539] Step 4:

[0540] The server generates an anomaly report based on the analysis results. The report includes the type and location of the detected anomaly, as well as risk information that requires immediate action if necessary. Furthermore, the server uses an emotion engine to analyze the user's current emotional data in order to recognize the user's emotional state.

[0541] Step 5:

[0542] After the emotion engine evaluates the user's emotional state, the server uses the evaluation results to notify the user of an anomaly in the most appropriate way. For example, if the user is experiencing stress, the server will respond by presenting the information in a simplified format.

[0543] Step 6:

[0544] Users view analysis reports from the server through a user interface. The interface displays anomalies on a map, and detailed information for each anomaly is provided in an easily accessible format.

[0545] Step 7:

[0546] Users plan appropriate actions based on the report. Specifically, they may dispatch a maintenance team or arrange for additional inspection work. They can also contribute to continuous improvement by providing feedback to the system.

[0547] (Example 2)

[0548] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] In modern infrastructure, early detection of anomalies and rapid and effective information provision to users are crucial. However, conventional systems lacked sufficient accuracy in data noise reduction, assessment of anomaly urgency, and appropriate information presentation methods tailored to users' emotional states. As a result, on-site responses were delayed, leading to increased user stress. This invention aims to solve these problems and support safe and efficient infrastructure operation.

[0550] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0551] In this invention, the server includes means for denoising and normalizing data, means for analyzing anomalies using a learning model and evaluating their urgency, and means for evaluating the user's emotional state using sentiment analysis technology and providing optimal information. This enables real-time anomaly detection and the provision of appropriate information to reduce user stress.

[0552] A "fluid pathway" is a passage or pipe through which a fluid, such as a liquid or gas, moves.

[0553] "Video data" refers to digital data containing visual information acquired using devices such as cameras.

[0554] "Vibration data" refers to information about vibrations of objects or the environment that are recorded using sensors.

[0555] "Acoustic data" refers to data collected by sound sensors that includes information about the surrounding sounds.

[0556] "Data processing means" refers to the system's functions, including the process of removing noise and normalizing acquired data.

[0557] A "learning model" is a model that analyzes data based on machine learning algorithms to perform pattern recognition and anomaly detection.

[0558] "Anomaly detection" is the process of identifying unusual data patterns or behaviors that indicate potential problems in a system.

[0559] "Emotional analysis technology" is a technology that analyzes voice and facial expression data to evaluate a user's emotional state.

[0560] A "user interface" refers to the screens and devices that a user uses to interact with a system and obtain information.

[0561] "Optimal information presentation" means providing information in a format that is easy to understand and reduces stress, tailored to the user's current emotional state.

[0562] The present invention is an advanced infrastructure management technology aimed at early detection of anomalies in fluid pathways and providing users with appropriate information. This system collects data using a group of devices installed in the fluid pathway, analyzes it using a server, and provides an interface for users to view the information.

[0563] terminal

[0564] The terminal moves along a fluid path and controls multiple sensors. Specifically, it is equipped with a camera that captures omnidirectional video, an accelerometer that detects vibrations, and a microphone that picks up sound. The terminal compresses and packages the data obtained from these sensors in real time and transmits it to the base station. A dedicated communication module is used for data transmission, enabling real-time communication.

[0565] server

[0566] The server receives data transmitted from the base station and first performs noise reduction. It cleanses and normalizes the data using programming libraries such as Python and NumPy. Next, it runs an analysis model on the processed data using machine learning libraries such as TensorFlow to detect anomalies and assess their urgency. Furthermore, it integrates the analysis results with sentiment analysis technology to evaluate the user's emotional state and provide appropriate information.

[0567] User

[0568] Users can view these analysis results through a user interface. This interface operates on a web browser and is designed to allow users to intuitively understand the anomalies and their details. Based on sentiment analysis technology, the format of information presentation is adjusted according to the user's current emotional state, playing a role in reducing stress and confusion.

[0569] As a concrete example, consider a case where abnormal vibrations are detected in a part of the fluid path. Based on this data, the server quickly identifies the anomaly and provides the user with a notification such as, "Abnormal vibration detected: Vibration confirmed in fluid path sector 3. No immediate action is required," thereby supporting the user in responding calmly.

[0570] The following are some possible prompt statements for a generative AI model.

[0571] "Summarize the latest anomaly analysis results in fluid pathways and propose an appropriate information presentation method to reduce user stress."

[0572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0573] Step 1:

[0574] The device moves through a fluid path, collecting data through sensors. Specifically, it captures omnidirectional video data with a camera, acquires vibration data with an accelerometer, and collects acoustic data with a microphone. The input is instantaneous environmental changes. This data is immediately compressed; for example, video data is encoded in H.264 format, and vibration and acoustic data in FLAC format. The output is compressed packaged data.

[0575] Step 2:

[0576] The terminal transmits the compressed data it has collected to the base station. The data is sent in real time via the LTE communication module. The input includes the data compressed in the previous step. Before transmission, the terminal adds a timestamp and device identification information to each packet of data to optimize it for transmission to the ground. The output is a data stream that can be received by the base station.

[0577] Step 3:

[0578] The server receives data from the base station and first processes it for denoising and normalization. The server uses Python to remove high-frequency noise from the received data and the NumPy library to normalize the data to a standard scale. The input is compressed data transmitted from the base station. The output is clean, analyzable data.

[0579] Step 4:

[0580] The server uses a machine learning model to analyze denoised data. The model, built using the TensorFlow library, detects anomalies in the data and identifies potential problems. The input is normalized data. This process identifies the type and urgency of the anomalies and generates warnings about potential risks. The output consists of the detected anomalies and their associated analysis results.

[0581] Step 5:

[0582] The user views the analysis results through a user interface. The user interface displays detailed analysis results and provides information optimized by the sentiment analysis engine. Inputs are analysis results from the server and the user's real-time sentiment data. Specifically, it displays anomalies on a map and summarizes information according to its urgency. Outputs are visual reports and status notifications that the user can review.

[0583] (Application Example 2)

[0584] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] Conventional infrastructure management systems, while capable of detecting and predicting anomalies in fluid pathways, lacked the ability to provide information tailored to the user's emotional state, resulting in insufficient improvements in work efficiency and stress reduction. Furthermore, when an anomaly actually occurs, users need real-time information and support to take appropriate action. However, existing systems have struggled to meet these needs.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes emotion recognition means for evaluating the user's emotional state and determining how to present information; means for notifying the user of information regarding the detection or prediction of anomalies visually and audibly; and means for a user interface that displays information on anomaly locations identified on a map, presents detailed analysis results, and provides stress reduction information tailored to the user's emotional state. This enables the user to access anomaly information in real time, reduce stress while responding efficiently, and improve work efficiency.

[0588] A "fluid pathway" is a path through which fluids such as liquids and gases move, and includes infrastructure structures such as pipelines and duct systems.

[0589] "Omnidirectional video data" refers to video data captured in all 360 degrees, recording the entire surroundings from a specific point.

[0590] "Vibration data" refers to data that shows the characteristics of vibrations obtained from fluid paths and the surrounding environment, and is information used to detect anomalies.

[0591] "Audio data" refers to data that shows acoustic signals collected from within or around a fluid path, and is useful for detecting anomalies and evaluating the environment.

[0592] A "learning model" is a computational method that uses machine learning algorithms to analyze data and identify patterns and anomalies.

[0593] "Emotion recognition means" refers to technology that analyzes a user's emotional state from their voice and facial expressions and generates appropriate feedback.

[0594] A "user interface" is a visual and operational interface for the exchange of information between the user and the system, and it presents analysis results and detailed information.

[0595] "Notification means" refers to visual or audio technologies used to inform users of information based on the detection or prediction of anomalies.

[0596] To implement this invention, it is necessary to construct a system for monitoring fluid paths and detecting anomalies. The terminal is integrated into a device such as smart glasses and collects omnidirectional video, vibration, and audio data while moving within the fluid path. This terminal transmits the collected data to a ground base station in real time. The hardware used would be smart glasses (e.g., HoloLens) and high-sensitivity sensors.

[0597] The server performs noise reduction and normalization on the received data and analyzes it using a learning model for anomaly detection. Based on this analysis, it detects anomalies and, if necessary, predicts signs of anomalies. Furthermore, it uses emotion recognition means to evaluate the user's emotional state from their voice and facial expressions and optimizes the information presentation method. The software used is an analysis program equipped with machine learning algorithms and an emotion recognition engine.

[0598] Through the user interface of the smart glasses, users can visually identify anomalies on a map and obtain detailed analysis results. Notifications visually and audibly alert the user to the presence or absence of anomalies, and stress reduction information tailored to their emotional state is provided. For example, if a pressure anomaly is detected during work, the glasses will display "Appropriate action is required regarding the detected pressure anomaly," and if stress is detected from the user's facial expression, an audible message will say, "It is important to proceed calmly in the next steps."

[0599] An example of a prompt message might be something like, "Consider providing information that is effective in detecting factory anomalies and reducing user stress." This allows the system to support users' efficient work and improve the overall security of infrastructure management.

[0600] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0601] Step 1:

[0602] The device moves through a fluid path while collecting omnidirectional video, vibration, and audio data in real time. In this step, smart glasses and sensors are used to convert the collected data into a digital format. The input here is physical video, vibration, and audio information, and the output is data in digital format.

[0603] Step 2:

[0604] The terminal collects digital data and transmits it to a ground base station in real time. Secure and high-speed data transfer is performed via a communication module. Inputs are digital video, vibration, and audio data, and output is the data transmission status to the base station.

[0605] Step 3:

[0606] The server performs noise reduction and normalization on the data received from the base station. This involves cleaning and scaling the data and converting it into a format suitable for analysis. The input is the data received from the base station, and the output is the processed, clean data.

[0607] Step 4:

[0608] The server uses a machine learning model to detect anomalies in processed data. Machine learning algorithms are used to identify anomaly patterns and predict future occurrences. The input is processed, clean data, and the output includes information on the presence or absence of anomalies and predictions.

[0609] Step 5:

[0610] The server uses emotion recognition to evaluate the user's emotional state. It analyzes voice and facial expression data to determine the user's psychological state. The input is the user's voice and facial expression data, and the output is the evaluation result of the emotional state.

[0611] Step 6:

[0612] Users view anomaly information and analysis results through smart glasses. The user interface visually displays anomalies on a map and provides details. Input is analysis results from the server, and output is visual and audio feedback to the user.

[0613] Step 7:

[0614] The server considers the user's emotional state and provides stress-reducing information and guidance. Using a generative AI model, it optimizes information presentation methods and generates prompts tailored to the user's needs. The input is the result of the emotional state assessment, and the output is the optimized information presentation.

[0615] This allows users to grasp anomaly information in real time and perform tasks more effectively.

[0616] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0617] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0618] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0619] [Fourth Embodiment]

[0620] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0621] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0622] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0623] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0624] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0625] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0626] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0627] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0628] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0629] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0632] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0633] The system of the present invention consists of a specially designed device and its control software, as well as a server program and user interface for data analysis, in order to efficiently detect anomalies in a fluid path. As an embodiment of this invention, the specific processing details of the program are described below in natural language.

[0634] terminal

[0635] First, the terminal controls a device placed within the fluid path. This device is equipped with a camera that captures omnidirectional video and various sensors that collect vibration and audio data. The terminal controls the device to move at an appropriate speed within the fluid path and simultaneously issues commands to capture data at regular intervals. This allows the data to be packaged and transmitted to the base station in real time.

[0636] server

[0637] The server receives data sent from the ground base station and prepares it for data analysis. The server program integrates this data and uses a specific learning model to identify patterns of abnormal operation. For example, if the server detects an abnormal sound, it analyzes its frequency components and predicts the need for future repairs based on the patterns identified as abnormal, and notifies the user accordingly.

[0638] User

[0639] Users can review the analysis results in detail through the provided user interface. The interface plots anomalies on a map of the fluid path and displays detailed information about each anomaly, helping users respond quickly and accurately. Based on this information, users can decide whether to dispatch a maintenance team or take countermeasures, and contribute to improving the accuracy of the system by providing feedback.

[0640] This embodiment of the invention enables significantly more efficient and precise monitoring and maintenance of fluid pathways than conventional methods.

[0641] The following describes the processing flow.

[0642] Step 1:

[0643] The terminal activates the devices placed within the fluid path. Once the devices are activated, the omnidirectional cameras and various sensors begin operating and perform initial calibration. After calibration is complete, the devices begin moving along the configured path at a constant speed.

[0644] Step 2:

[0645] The terminal instructs the moving device to periodically collect omnidirectional video and sensor data. The device generates data in packages at specified time intervals, making them available for the terminal to receive. These packages contain video, vibration, and audio data along with timestamps.

[0646] Step 3:

[0647] The terminal transmits the collected data to a ground base station in real time. The data is transmitted wirelessly and received by a server. The terminal also adds location information to each data package, allowing the collection point of each data to be identified.

[0648] Step 4:

[0649] The server receives data packages transmitted from ground base stations, verifies their integrity, and then prepares them for analysis. First, the server filters and denoises the data to improve processing efficiency, and then converts it into a format suitable for the learning model.

[0650] Step 5:

[0651] The server inputs the adjusted data into a learning model and executes the anomaly detection process. Specifically, it detects anomaly patterns and identifies predicted risks. Once the analysis results are obtained, they are compiled and saved as a report.

[0652] Step 6:

[0653] Users access analysis reports generated from the server through a user interface. The interface displays each anomaly on a map and presents detailed information about the detected locations. This allows users to take immediate corrective action.

[0654] Step 7:

[0655] Users decide on actions to take to address any anomalies they discover, such as dispatching a maintenance team as needed. Users can also feed back field-tested data to the server, which contributes to improving the overall accuracy of the system.

[0656] (Example 1)

[0657] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0658] Early detection of anomalies in fluid pathways is crucial, but current technologies often delay detection, potentially leading to serious problems. Furthermore, when anomalies are detected, accurate location information and detailed analysis results are necessary, requiring efficient and sophisticated technology. Therefore, there is a need for the development of new systems that enable rapid and accurate anomaly detection, prediction, and information provision.

[0659] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0660] In this invention, the server includes means for integrating and analyzing omnidirectional video information, vibration information, and audio information within a fluid path; means for detecting anomalies and predicting signs of anomaly occurrence based on the analysis results; means for transmitting analysis information in real time via a communication network; and means for providing a human-machine interface for visualizing the analyzed information and identifying the location of anomalies. This enables rapid and accurate anomaly detection and prediction, as well as effective response through the provision of more detailed information.

[0661] A "fluid pathway" refers to a passage through which a liquid or gas moves, and includes pipelines and ducts.

[0662] "Omnidirectional video information" refers to video data that can capture the entire 360-degree field of view, encompassing all visual information of the surroundings from a specific location.

[0663] "Vibration information" refers to data on the physical vibrations of objects and structures, and is information on periodic motion measured by sensors.

[0664] "Audio information" refers to data that describes the characteristics and properties of sound, and is obtained through the capture of sound signals by sensors.

[0665] An "information analysis device" refers to a device that receives various types of information as input, integrates and analyzes them, and outputs the results.

[0666] A "learning algorithm" refers to a method and procedure for learning patterns based on input data and performing analysis or prediction.

[0667] A "human-machine interface" is a means and interface for a user to interact with a machine, and includes elements that enable the visualization and input of information.

[0668] A "communication network" refers to a system for transmitting and sharing information, and includes means for connecting computers and other devices.

[0669] "Anomaly detection" refers to the process or ability to identify a state that deviates from normal operation.

[0670] "Predictive forecasting" refers to analysis and prediction aimed at foreseeing potential future changes or anomalies.

[0671] The system of this invention is designed to efficiently detect anomalies in fluid pathways. Specific embodiments of each component are shown below.

[0672] terminal

[0673] The terminal controls sensor-equipped devices placed within the fluid path. These devices are equipped with cameras for capturing omnidirectional video information and various sensors for acquiring vibration and audio information. The terminal captures and packages data in real time as the devices move within the fluid path. The collected data is transmitted to a ground base station via a communication network. Typical communication technologies used include Wi-Fi and mobile data communication.

[0674] server

[0675] The server is a device that receives and analyzes data transmitted through ground base stations. The server is equipped with a learning algorithm that identifies anomaly patterns based on the integrated data. This allows the server to perform anomaly detection in real time. For example, generative AI models are used to recognize abnormal vibrations and sound patterns during the analysis. If an anomaly is detected, the server predicts its precursors and notifies the user of the necessary information.

[0676] User

[0677] Users can review the system's analysis results and obtain detailed information about anomalies. Through a dedicated human-machine interface, users can access detailed information, including data visualizations, enabling rapid response. The interface allows users to issue maintenance instructions and arrange for repairs to anomalies. Furthermore, the feedback received can be used to improve the system.

[0678] For example, if an abnormal sound pattern is detected within the fluid path, the location will be displayed in red on the map, and detailed frequency analysis data will be provided. Furthermore, information will be provided to the user through prompt messages such as, "Analyze the sound and vibration data collected by devices installed within the fluid path, and notify us if any abnormalities are found. Display the location of the abnormality on the map so that maintenance can be arranged as needed."

[0679] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0680] Step 1:

[0681] The terminal activates sensor-equipped devices placed within the fluid path to collect omnidirectional video, vibration, and audio information. The input here is raw data from each sensor, which the terminal converts into a digital format and packages. This data package is transmitted to a ground base station via Wi-Fi or a mobile data network. Specifically, the device captures data every second as it moves and transmits it at regular time intervals.

[0682] Step 2:

[0683] The server decompresses the data packages received from the ground base station and verifies the data's integrity. The input is packaged data transmitted from the terminal, and the output is integrated data in a format suitable for analysis. The server sorts this data based on a time axis and prepares it for anomaly detection. During this process, it inserts an operation to perform predictive imputation if there is missing data.

[0684] Step 3:

[0685] The server performs analysis using a generative AI model with the integrated data. The input here is the prepared data obtained in step 2, and the output is the anomaly detection and prediction results. The server uses a learning algorithm to extract features from the data and identify anomaly patterns. If an anomaly is recognized, it predicts future anomaly occurrences based on those features. For example, it analyzes the frequency components of audio data and compares them with past anomaly cases.

[0686] Step 4:

[0687] The server converts the analysis results into a format understandable to the user and presents them through a dedicated human-machine interface. The input is the analysis results obtained in step 3, and the output is visualized information and specific recommendations. For example, it can plot anomalies on a fluid path map and display the specific causes of the anomalies and suggested countermeasures.

[0688] Step 5:

[0689] Users make quick and accurate decisions based on information obtained through the interface. The input is analytical information provided by the server, which users evaluate and issue maintenance instructions. Providing feedback also contributes to further system improvements. Specific actions include clicking on anomalies to view detailed information and sending instructions to the maintenance team via email.

[0690] (Application Example 1)

[0691] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] It is crucial to detect abnormalities in fluid pathways early and prevent unexpected failures of machinery and equipment. However, predicting abnormalities is currently difficult, requiring rapid and precise monitoring and judgment. Furthermore, there is a lack of visual information to smoothly identify abnormal locations and instruct corrective actions.

[0693] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0694] In this invention, the server includes information analysis means that include a model for integrating and analyzing omnidirectional visual data, vibration data, and acoustic data; means for detecting abnormalities and predicting signs of anomaly occurrence; means for transmitting sequential information via wireless communication; and means for visualizing the results of anomaly detection based on time and location information. This enables early detection of anomalies in the fluid path, accurate location identification, and rapid implementation of countermeasures.

[0695] A "device that moves within a fluid passage" is a device that moves within a fluid passage while acquiring information about its surroundings.

[0696] "Omnidirectional visual data" refers to data that includes video information of the entire surrounding area at a given point.

[0697] "Vibration data" refers to data that includes information about vibrations within a fluid passage.

[0698] "Acoustic data" refers to data that contains information related to sounds generated within a fluid passage.

[0699] An "information analysis device" is a device designed to analyze collected data.

[0700] A "model" is a computational method or algorithm used to identify a specific pattern.

[0701] "Abnormal" is a term that refers to an abnormal state that deviates from the normal range.

[0702] "Sequential information" is a term that refers to data that is collected and transmitted immediately.

[0703] "Wireless communication" is a method of transmitting information over long distances using radio waves.

[0704] "Visualization" refers to displaying data and information in a way that is easy for people to understand.

[0705] "Time and location information" refers to data that includes information about a geographical location at a specific point in time.

[0706] The system implementing this invention provides a comprehensive function for monitoring fluid pathways within a factory. Its elements and operating processes are described below.

[0707] The terminal moves within the fluid path and acquires data through multi-directionally positioned visual and acoustic sensors. These sensors are used to collect omnidirectional visual data, vibration data, and acoustic data. This data is continuously transmitted from the terminal to the server via wireless communication.

[0708] The server is the central device that receives the collected data and performs information analysis. The server has an information analysis system built in a Python environment and is equipped with an advanced computational model using TensorFlow. This model identifies anomaly patterns and executes algorithms to predict the likelihood of anomaly occurrences early on. Furthermore, if an anomaly is detected, it identifies its location based on the data and performs visualization processing based on time and location information.

[0709] Users can view detailed analysis results through an administration screen designed with React.js. This user interface clearly displays the location of detected anomalies on a map, helping users quickly identify problems and take appropriate action.

[0710] As a concrete example, when introducing new machinery to a factory, this system can be used to monitor the fluid pathways, allowing for the rapid detection of problems that tend to occur in the initial stages of implementation, thereby preventing major failures. By utilizing a generative AI model, for example, by inputting a prompt such as, "If abnormal vibrations or sounds are detected in the fluid pathways of newly introduced machinery, please list their locations and countermeasures," it is possible to compare them with known abnormality patterns and suggest appropriate countermeasures.

[0711] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0712] Step 1:

[0713] The terminal collects omnidirectional visual data, vibration data, and acoustic data from multidirectional sensors installed within the fluid channel. The input is real-world environmental information within the fluid channel, and the output is digital data that packages this data together. This data is acquired directly from the sensor devices and processed in real time.

[0714] Step 2:

[0715] The terminal transmits the collected digital data to the server via wireless communication. The input is the digital data packaged in step 1, and the output is the data received by the server. Specifically, the Wi-Fi module manages the data transmission and applies the appropriate protocol to prevent data loss.

[0716] Step 3:

[0717] The server feeds the received digital data into an analysis model to identify anomalous patterns. The input is the digital data received by the server, and the output is the analysis results of the patterns and signs of anomalies that have been determined to be anomalous. TensorFlow is used for this analysis, and the specific computational processes for anomaly detection are performed using machine learning algorithms.

[0718] Step 4:

[0719] The server identifies the location of the anomaly based on the analysis results, adds time and location information, and generates visualization data. The input is the analysis results obtained in step 3, and the output is the visualized anomaly information. Specifically, it integrates map information and analysis results, and highlights the necessary information using a visualization algorithm.

[0720] Step 5:

[0721] Users review visualized anomaly information through the management screen and determine the necessary countermeasures. The input is the visualized data provided in step 4, and the output is the countermeasures and action plan decided by the user. Specifically, the user interface effectively indicates the location of the anomaly and assists the user in making quick decisions.

[0722] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0723] The present invention provides an advanced infrastructure management system that, in addition to monitoring fluid pathways and detecting anomalies, also has the ability to recognize the emotional state of the user. An embodiment of the system is configured as follows:

[0724] terminal

[0725] The terminal controls a device installed within the fluid path. This device has a set of sensors to capture omnidirectional video and collect vibration and audio data. The device moves along the fluid path, packaging the data collected by the terminal and transmitting it in real time to a ground base station.

[0726] server

[0727] The server processes data received from the base station. The received dataset undergoes predetermined noise reduction and data normalization before being analyzed by a learning model. Based on the analysis results, the server quickly detects anomalies, predicts potential risks, and takes appropriate action. Furthermore, this server program incorporates an emotion engine to evaluate user reactions based on the analysis results.

[0728] User

[0729] Users can view anomaly detection reports in real time through a user interface that displays data analysis information. The user interface visually shows detected anomalies on a map and provides detailed reports. Furthermore, the emotion engine analyzes the user's voice and facial expressions to automatically select the optimal information presentation method. This plays a role in increasing efficiency in the user's daily work and supporting accurate maintenance tasks.

[0730] As a concrete example, when users encounter complex anomaly analysis results, the emotion engine summarizes and presents the information to reduce their stress levels. For instance, if the urgency of an anomaly is judged to be low, the notification can be presented in simple language to avoid confusing the user.

[0731] This embodiment enables the system to adapt to environmental changes, improve the user experience, and achieve secure and efficient infrastructure management.

[0732] The following describes the processing flow.

[0733] Step 1:

[0734] The terminal activates the device placed within the fluid path and prepares to capture omnidirectional video. After activation, the device uses its omnidirectional camera and various sensors to collect video, vibration, and audio data as it moves along the designated route.

[0735] Step 2:

[0736] The terminal packages the collected data in real time and then transmits it to a ground base station. The package includes all the collected data as well as location information, which allows the data collection point to be identified.

[0737] Step 3:

[0738] The server receives data from the ground base station and preprocesses it by performing noise reduction and data filtering. The preprocessed data is then fed into a learning model to perform anomaly detection and risk prediction.

[0739] Step 4:

[0740] The server generates an anomaly report based on the analysis results. The report includes the type and location of the detected anomaly, as well as risk information that requires immediate action if necessary. Furthermore, the server uses an emotion engine to analyze the user's current emotional data in order to recognize the user's emotional state.

[0741] Step 5:

[0742] After the emotion engine evaluates the user's emotional state, the server uses the evaluation results to notify the user of an anomaly in the most appropriate way. For example, if the user is experiencing stress, the server will respond by presenting the information in a simplified format.

[0743] Step 6:

[0744] Users view analysis reports from the server through a user interface. The interface displays anomalies on a map, and detailed information for each anomaly is provided in an easily accessible format.

[0745] Step 7:

[0746] Users plan appropriate actions based on the report. Specifically, they may dispatch a maintenance team or arrange for additional inspection work. They can also contribute to continuous improvement by providing feedback to the system.

[0747] (Example 2)

[0748] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] In modern infrastructure, early detection of anomalies and rapid and effective information provision to users are crucial. However, conventional systems lacked sufficient accuracy in data noise reduction, assessment of anomaly urgency, and appropriate information presentation methods tailored to users' emotional states. As a result, on-site responses were delayed, leading to increased user stress. This invention aims to solve these problems and support safe and efficient infrastructure operation.

[0750] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0751] In this invention, the server includes means for denoising and normalizing data, means for analyzing anomalies using a learning model and evaluating their urgency, and means for evaluating the user's emotional state using sentiment analysis technology and providing optimal information. This enables real-time anomaly detection and the provision of appropriate information to reduce user stress.

[0752] A "fluid pathway" is a passage or pipe through which a fluid, such as a liquid or gas, moves.

[0753] "Video data" refers to digital data containing visual information acquired using devices such as cameras.

[0754] "Vibration data" refers to information about vibrations of objects or the environment that are recorded using sensors.

[0755] "Acoustic data" refers to data collected by sound sensors that includes information about the surrounding sounds.

[0756] "Data processing means" refers to the system's functions, including the process of removing noise and normalizing acquired data.

[0757] A "learning model" is a model that analyzes data based on machine learning algorithms to perform pattern recognition and anomaly detection.

[0758] "Anomaly detection" is the process of identifying unusual data patterns or behaviors that indicate potential problems in a system.

[0759] "Emotional analysis technology" is a technology that analyzes voice and facial expression data to evaluate a user's emotional state.

[0760] A "user interface" refers to the screens and devices that a user uses to interact with a system and obtain information.

[0761] "Optimal information presentation" means providing information in a format that is easy to understand and reduces stress, tailored to the user's current emotional state.

[0762] The present invention is an advanced infrastructure management technology aimed at early detection of anomalies in fluid pathways and providing users with appropriate information. This system collects data using a group of devices installed in the fluid pathway, analyzes it using a server, and provides an interface for users to view the information.

[0763] terminal

[0764] The terminal moves along a fluid path and controls multiple sensors. Specifically, it is equipped with a camera that captures omnidirectional video, an accelerometer that detects vibrations, and a microphone that picks up sound. The terminal compresses and packages the data obtained from these sensors in real time and transmits it to the base station. A dedicated communication module is used for data transmission, enabling real-time communication.

[0765] server

[0766] The server receives data transmitted from the base station and first performs noise reduction. It cleanses and normalizes the data using programming libraries such as Python and NumPy. Next, it runs an analysis model on the processed data using machine learning libraries such as TensorFlow to detect anomalies and assess their urgency. Furthermore, it integrates the analysis results with sentiment analysis technology to evaluate the user's emotional state and provide appropriate information.

[0767] User

[0768] Users can view these analysis results through a user interface. This interface operates on a web browser and is designed to allow users to intuitively understand the anomalies and their details. Based on sentiment analysis technology, the format of information presentation is adjusted according to the user's current emotional state, playing a role in reducing stress and confusion.

[0769] As a concrete example, consider a case where abnormal vibrations are detected in a part of the fluid path. Based on this data, the server quickly identifies the anomaly and provides the user with a notification such as, "Abnormal vibration detected: Vibration confirmed in fluid path sector 3. No immediate action is required," thereby supporting the user in responding calmly.

[0770] The following are some possible prompt statements for a generative AI model.

[0771] "Summarize the latest anomaly analysis results in fluid pathways and propose an appropriate information presentation method to reduce user stress."

[0772] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0773] Step 1:

[0774] The device moves through a fluid path, collecting data through sensors. Specifically, it captures omnidirectional video data with a camera, acquires vibration data with an accelerometer, and collects acoustic data with a microphone. The input is instantaneous environmental changes. This data is immediately compressed; for example, video data is encoded in H.264 format, and vibration and acoustic data in FLAC format. The output is compressed packaged data.

[0775] Step 2:

[0776] The terminal transmits the compressed data it has collected to the base station. The data is sent in real time via the LTE communication module. The input includes the data compressed in the previous step. Before transmission, the terminal adds a timestamp and device identification information to each packet of data to optimize it for transmission to the ground. The output is a data stream that can be received by the base station.

[0777] Step 3:

[0778] The server receives data from the base station and first processes it for denoising and normalization. The server uses Python to remove high-frequency noise from the received data and the NumPy library to normalize the data to a standard scale. The input is compressed data transmitted from the base station. The output is clean, analyzable data.

[0779] Step 4:

[0780] The server uses a machine learning model to analyze denoised data. The model, built using the TensorFlow library, detects anomalies in the data and identifies potential problems. The input is normalized data. This process identifies the type and urgency of the anomalies and generates warnings about potential risks. The output consists of the detected anomalies and their associated analysis results.

[0781] Step 5:

[0782] The user views the analysis results through a user interface. The user interface displays detailed analysis results and provides information optimized by the sentiment analysis engine. Inputs are analysis results from the server and the user's real-time sentiment data. Specifically, it displays anomalies on a map and summarizes information according to its urgency. Outputs are visual reports and status notifications that the user can review.

[0783] (Application Example 2)

[0784] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] Conventional infrastructure management systems, while capable of detecting and predicting anomalies in fluid pathways, lacked the ability to provide information tailored to the user's emotional state, resulting in insufficient improvements in work efficiency and stress reduction. Furthermore, when an anomaly actually occurs, users need real-time information and support to take appropriate action. However, existing systems have struggled to meet these needs.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0787] In this invention, the server includes emotion recognition means for evaluating the user's emotional state and determining how to present information; means for notifying the user of information regarding the detection or prediction of anomalies visually and audibly; and means for a user interface that displays information on anomaly locations identified on a map, presents detailed analysis results, and provides stress reduction information tailored to the user's emotional state. This enables the user to access anomaly information in real time, reduce stress while responding efficiently, and improve work efficiency.

[0788] A "fluid pathway" is a path through which fluids such as liquids and gases move, and includes infrastructure structures such as pipelines and duct systems.

[0789] "Omnidirectional video data" refers to video data captured in all 360 degrees, recording the entire surroundings from a specific point.

[0790] "Vibration data" refers to data that shows the characteristics of vibrations obtained from fluid paths and the surrounding environment, and is information used to detect anomalies.

[0791] "Audio data" refers to data that shows acoustic signals collected from within or around a fluid path, and is useful for detecting anomalies and evaluating the environment.

[0792] A "learning model" is a computational method that uses machine learning algorithms to analyze data and identify patterns and anomalies.

[0793] "Emotion recognition means" refers to technology that analyzes a user's emotional state from their voice and facial expressions and generates appropriate feedback.

[0794] A "user interface" is a visual and operational interface for the exchange of information between the user and the system, and it presents analysis results and detailed information.

[0795] "Notification means" refers to visual or audio technologies used to inform users of information based on the detection or prediction of anomalies.

[0796] To implement this invention, it is necessary to construct a system for monitoring fluid paths and detecting anomalies. The terminal is integrated into a device such as smart glasses and collects omnidirectional video, vibration, and audio data while moving within the fluid path. This terminal transmits the collected data to a ground base station in real time. The hardware used would be smart glasses (e.g., HoloLens) and high-sensitivity sensors.

[0797] The server performs noise reduction and normalization on the received data and analyzes it using a learning model for anomaly detection. Based on this analysis, it detects anomalies and, if necessary, predicts signs of anomalies. Furthermore, it uses emotion recognition means to evaluate the user's emotional state from their voice and facial expressions and optimizes the information presentation method. The software used is an analysis program equipped with machine learning algorithms and an emotion recognition engine.

[0798] Through the user interface of the smart glasses, users can visually identify anomalies on a map and obtain detailed analysis results. Notifications visually and audibly alert the user to the presence or absence of anomalies, and stress reduction information tailored to their emotional state is provided. For example, if a pressure anomaly is detected during work, the glasses will display "Appropriate action is required regarding the detected pressure anomaly," and if stress is detected from the user's facial expression, an audible message will say, "It is important to proceed calmly in the next steps."

[0799] An example of a prompt message might be something like, "Consider providing information that is effective in detecting factory anomalies and reducing user stress." This allows the system to support users' efficient work and improve the overall security of infrastructure management.

[0800] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0801] Step 1:

[0802] The device moves through a fluid path while collecting omnidirectional video, vibration, and audio data in real time. In this step, smart glasses and sensors are used to convert the collected data into a digital format. The input here is physical video, vibration, and audio information, and the output is data in digital format.

[0803] Step 2:

[0804] The terminal collects digital data and transmits it to a ground base station in real time. Secure and high-speed data transfer is performed via a communication module. Inputs are digital video, vibration, and audio data, and output is the data transmission status to the base station.

[0805] Step 3:

[0806] The server performs noise reduction and normalization on the data received from the base station. This involves cleaning and scaling the data and converting it into a format suitable for analysis. The input is the data received from the base station, and the output is the processed, clean data.

[0807] Step 4:

[0808] The server uses a machine learning model to detect anomalies in processed data. Machine learning algorithms are used to identify anomaly patterns and predict future occurrences. The input is processed, clean data, and the output includes information on the presence or absence of anomalies and predictions.

[0809] Step 5:

[0810] The server uses emotion recognition to evaluate the user's emotional state. It analyzes voice and facial expression data to determine the user's psychological state. The input is the user's voice and facial expression data, and the output is the evaluation result of the emotional state.

[0811] Step 6:

[0812] Users view anomaly information and analysis results through smart glasses. The user interface visually displays anomalies on a map and provides details. Input is analysis results from the server, and output is visual and audio feedback to the user.

[0813] Step 7:

[0814] The server considers the user's emotional state and provides stress-reducing information and guidance. Using a generative AI model, it optimizes information presentation methods and generates prompts tailored to the user's needs. The input is the result of the emotional state assessment, and the output is the optimized information presentation.

[0815] This allows users to grasp anomaly information in real time and perform tasks more effectively.

[0816] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0817] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0819] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0820] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0821] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0822] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0823] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0824] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0825] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0826] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0827] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0828] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0829] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0830] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0831] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0832] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0833] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0834] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0835] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0836] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0837] The following is further disclosed regarding the embodiments described above.

[0838] (Claim 1)

[0839] A means for collecting omnidirectional video data while moving within a fluid path,

[0840] Means for collecting vibration and sound data within the fluid path,

[0841] A data analysis means including a learning model that synthesizes and analyzes the aforementioned omnidirectional video data, vibration data, and audio data,

[0842] A means for detecting anomalies based on the aforementioned analysis results and predicting signs of anomaly occurrence,

[0843] A means for transmitting analysis data in real time via communication with a ground base station,

[0844] A system that includes this.

[0845] (Claim 2)

[0846] The system according to claim 1, wherein the learning model uses a machine learning algorithm for identifying abnormal patterns.

[0847] (Claim 3)

[0848] The system according to claim 1, comprising a user interface that displays information on anomaly locations identified on a map, and means for presenting detailed analysis results via the user interface.

[0849] "Example 1"

[0850] (Claim 1)

[0851] A device that moves within a fluid path while collecting omnidirectional video information,

[0852] A device for collecting vibration and sound information within the fluid path,

[0853] An information analysis device including a learning algorithm that integrates and analyzes the aforementioned omnidirectional video information, vibration information, and audio information,

[0854] A device that detects abnormalities based on the aforementioned analysis results and predicts signs of an abnormality occurring,

[0855] A device that transmits analysis information in real time via a communication network,

[0856] A device equipped with a human-machine interface for visualizing analyzed information and identifying abnormal areas,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, wherein the learning algorithm uses a machine learning technique for identifying abnormal patterns.

[0860] (Claim 3)

[0861] The system according to claim 1, comprising a user interface that provides information on anomaly locations identified on a map, and means that provide detailed analysis results via the user interface to enable a rapid response.

[0862] "Application Example 1"

[0863] (Claim 1)

[0864] A device that moves within a fluid path while collecting omnidirectional visual data,

[0865] A device for collecting vibration and acoustic data within the fluid passage,

[0866] An information analysis device including a model that integrates and analyzes the aforementioned omnidirectional visual data, vibration data, and acoustic data,

[0867] A device that detects abnormalities based on the aforementioned analysis results and predicts signs of an anomaly,

[0868] A device that transmits information sequentially via wireless communication,

[0869] A device that visualizes the results of anomaly detection based on time and location information,

[0870] A system that includes this.

[0871] (Claim 2)

[0872] The system according to claim 1, wherein the model uses an algorithm for identifying abnormal patterns.

[0873] (Claim 3)

[0874] The system according to claim 1, comprising a management screen that presents visual information, and providing detailed analysis results via the management screen.

[0875] "Example 2 of combining an emotion engine"

[0876] (Claim 1)

[0877] A means of collecting video data while moving within a fluid path,

[0878] Means for collecting vibration and acoustic data within the fluid path,

[0879] A data processing means that processes the aforementioned omnidirectional video data, vibration data, and audio data, performs noise reduction and normalization, and analyzes the data using a learning model,

[0880] A means for detecting anomalies based on the aforementioned analysis results and predicting the urgency and signs of the anomalies,

[0881] A means of evaluating a user's emotional state using emotion analysis technology and providing the optimal method for presenting information,

[0882] A means for transmitting analysis data in real time via communication with a ground base station,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, wherein the learning model uses a machine learning algorithm to identify anomaly patterns and predict potential anomaly risks.

[0886] (Claim 3)

[0887] The system according to claim 1, comprising a user interface that displays information on anomaly locations identified on a map, a means for presenting detailed analysis results via the user interface, and a means for optimizing the information presentation using sentiment analysis technology.

[0888] "Application example 2 when combining with an emotional engine"

[0889] (Claim 1)

[0890] A means for collecting omnidirectional video data while moving within a fluid path,

[0891] Means for collecting vibration and sound data within the fluid path,

[0892] A data analysis means including a learning model that synthesizes and analyzes the aforementioned omnidirectional video data, vibration data, and audio data,

[0893] A means for detecting anomalies based on the aforementioned analysis results and predicting signs of anomaly occurrence,

[0894] An emotion recognition means for evaluating the user's emotional state and determining how to present information,

[0895] Means for notifying the user of information regarding the detection or prediction of anomalies visually and audibly,

[0896] A means for transmitting analysis data in real time via communication with a ground base station,

[0897] A system that includes this.

[0898] (Claim 2)

[0899] The system according to claim 1, wherein the learning model uses a machine learning algorithm for identifying abnormal patterns and combines it with an emotion recognition engine.

[0900] (Claim 3)

[0901] The system according to claim 1, comprising a user interface that displays information on anomaly locations identified on a map, a means for presenting detailed analysis results via the user interface, and a means for providing stress reduction information according to the user's emotional state. [Explanation of Symbols]

[0902] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting omnidirectional video data while moving within a fluid path, Means for collecting vibration and sound data within the fluid path, A data analysis means including a learning model that synthesizes and analyzes the aforementioned omnidirectional video data, vibration data, and audio data, A means for detecting anomalies based on the aforementioned analysis results and predicting signs of anomaly occurrence, A means for transmitting analysis data in real time via communication with a ground base station, A system that includes this.

2. The system according to claim 1, wherein the learning model uses a machine learning algorithm for identifying abnormal patterns.

3. The system according to claim 1, comprising a user interface that displays information on anomaly locations identified on a map, and means for presenting detailed analysis results via the user interface.

Images