On-device hmi edge device system having integrated real-time fire prediction and robot anomaly detection functions based on multi-sensor fusion physical ai and a method for operating the same

Abstract

The present invention relates to an on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, and a method for operating the same. More specifically, it relates to an on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, which completely eliminates the risk of equipment malfunction and process interruption caused by external thermal environment noise, prevents production interruption losses, can detect fire precursor symptoms at an extremely early stage using a precision sensor fusion method compared to a single sensor, and maximizes safety, and a method for operating the same.

Description

Technology Field

[0001] The present invention relates to an on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, and a method for operating the same. More specifically, by comparing and normalizing in real-time the deviation between a gradual rise in ambient temperature due to seasonal fluctuations or ventilation facility conditions and a rapid rise in local hot points (IR array), such as specific circuit breaker areas, it completely eliminates the risk of equipment malfunction and process stoppage caused by external thermal environment noise. Furthermore, it prevents production stoppage losses by perfectly distinguishing 'simple driving / motion loads' within the normal range—not only for floor-mounted robots but also for mobile robots with complex drive forms involving AMR / AGV driving inertia or rail transport vibration—through the analysis of movement trajectory profiles and the verification of combined spatial density of environmental data. Additionally, by integrating heterogeneous multimodal sensors, such as ultrasonic arc, gas, and flame sensors, along with dual temperature feature vectors using Gaussian Mixture Model (GMM)-based spatial density computation, it can detect fire precursor symptoms at the very early stages using a precise sensor fusion method compared to a single sensor. Moreover, the HMI deployed at the site does not rely on the backbone network of an external cloud or a central control server. The present invention relates to an on-device HMI edge device system having integrated real-time fire prediction and robot anomaly detection functions based on multi-sensor fusion physical AI, which processes inference and control command issuance in a one-stop manner within the edge device itself, and maximizes safety by simultaneously operating a physical hardware relay power cut-off circuit and a safe deceleration stop through the modification of the robot's distributed communication message upon confirmation of an emergency accident. Background Technology

[0002] In the prior art related to the present invention, Patent Document 1 describes a fire indication gas measuring device comprising a chamber for arranging a fire indication display target formed in a conductor part, a conductor part formed inside the chamber, a fire indication display target formed in the conductor part, a heating part for heating the conductor part, a temperature measuring part for measuring the temperature of the conductor part, and a gas measuring part for measuring the concentration of gas generated from the fire indication display target by the heat of the conductor part, thereby measuring the concentration of gas generated from the fire indication display target. Additionally, Patent Document 2 describes an internal temperature monitoring and control system for a power conversion device that can detect parts vulnerable to fire through a heat distribution sensor. Furthermore, Patent Document 3 describes an arc and fire detection system for a modularized power station that can cut off the power to the inverter, cut off the circuit connection between the inverter and the switchboard, cut off the power to the low-voltage panel, or cut off the power to the high-voltage panel based on the arc and the temperature of the power station detected by the control part of the power station. In addition, the switchgear and the method for handling abnormal signs of the switchgear, which are equipped with a function to detect abnormal signs in Patent Document 4, detect current leakage, arc, fire, and vibration caused by earthquakes, and detect abnormal signs that can be prepared for.

[0003] With the recent rapid technological advancements in smart factories and manufacturing process automation, the introduction of various types of articulated robots has become an essential element. The scope of application for these articulated robots is expanding beyond fixed articulated robotic arms that are fixed to the process floor to perform repetitive tasks, to mobile articulated robots in the form of mobile manipulators mounted on top of autonomous mobile robots (AMRs) or automated guided vehicles (AGVs) to move across wide process areas and perform variable tasks, and even to gantry-type robots that are transported along rails built on ceilings or walls. However, in these high-risk manufacturing sites, large-scale fires and physical malfunction accidents are repeatedly occurring due to equipment aging, insulation breakdown caused by continuous operation and load, and collisions, resulting in massive loss of life and property.

[0004] In particular, one of the major warning signs of a fire occurring within a switchboard (or distribution board), which is a core facility supplying power to a factory, is a "micro-electric arc" and localized hot spots on specific components (e.g., circuit breakers, terminal blocks, etc.). Conventional fire detection systems merely monitor the overall ambient temperature inside the switchboard or trigger an alarm only after a critical temperature is exceeded; consequently, they fail to block the degradation occurring at the very beginning of specific hot spots, often resulting in the loss of the golden time for initial suppression. Conversely, there was a limitation in that if the ambient temperature inside the switchboard rose due to external environmental factors—such as the overall temperature increase inside the factory during the summer or the temporary shutdown of ventilation systems—the system would generate false alarms even when there was no actual fire, unnecessarily halting the production process.

[0005] Furthermore, in the case of conventional technology that monitors only the motor current of an industrial robot to prevent abnormal behavior or collision accidents, there was a problem of reduced productivity where the entire process was unnecessarily halted due to the current threshold being temporarily exceeded, causing false alarms even during normal process load conditions where the robot accelerates rapidly or transports heavy loads. In particular, for mobile robots that repeatedly drive and stop or perform acceleration and deceleration on rails in combination with AMR / AGVs, rapid surge fluctuations in motor current, drive voltage, and joint torque are constantly induced even during normal operation due to movement inertia and physical vibration variations, making it structurally impossible to block false alarms using a simple threshold comparison method for motor current.

[0006] In addition, existing systems adopt an architecture that transmits collected data to a remote central control server or cloud for analysis, which causes network latency due to data transmission and has a fatal limitation in that it cannot immediately control the robot (e.g., ultra-low latency control within 0.5 seconds) in the event of an emergency, thereby failing to prevent physical collisions or the spread of destruction.

[0007] Therefore, there is an urgent need to develop a Physical AI-based on-device integrated platform that can autonomously stop equipment within 0.5 seconds in the event of an actual fire or abnormal behavior, while fundamentally blocking false alarms caused by environmental noise and robot driving / motion inertia, by separating and measuring ambient temperature data representing the overall temperature inside the switchboard and IR array temperature data that non-contactually scans local hot spots of specific key parts at the HMI edge device at the site without the intervention of an external server, and performing complex calculations with multidimensional dynamic feedback data acquired in real time regardless of the robot's kinematic form (fixed, mobile, rail type, etc.). Prior art literature

[0008] Registered Patent Publication No. 10-2499460 Registered Patent Publication No. 10-2748892 Registered Patent Publication No. 10-2861783 Registered Patent Publication No. 10-2933851 The problem to be solved

[0009] The present invention aims to provide an on-device HMI edge device system having an integrated function for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, which predicts the occurrence of fire in a switchboard and a multi-joint robot, detects anomalies, and decelerates and stops the multi-joint robot and cuts off the physical power supply, and a method for operating the same. means of solving the problem

[0010] An on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to a preferred embodiment of the present invention comprises: a data collection and linkage module that measures surrounding environment and fire sign data and acquires multidimensional robot dynamic feedback data; a multidimensional feature vector extraction and preprocessing module that normalizes heterogeneous time series data collected from the data collection and linkage module and refines multidimensional feature vectors by matching timestamps based on a specific repetitive process cycle or a movement drive cycle between specific points of a multi-joint robot; a machine learning-based intelligent anomaly detection engine that calculates a real-time likelihood-related reliability value of a multidimensional feature vector output from the multidimensional feature vector extraction and preprocessing module to distinguish between a temporary simple load state and an actual accident state; and a control unit that performs an autonomous safety control and output module that controls the operation of a multi-joint robot and visualizes emergency information when the accident state is finally confirmed by the machine learning-based intelligent anomaly detection engine (300).

[0011] In addition, the data collection and interlocking module is characterized by including: an ambient temperature sensor for measuring the air temperature and ambient temperature inside the distribution panel; a composite sensor module that non-contactually scans specific hazardous areas of circuit breakers or terminal blocks where loads are concentrated within the distribution panel to generate grid-type area temperature data, measures arc discharge shock wave acoustics, measures off-gas, and measures the unique wavelength band of a flame; and a robot data acquisition module that interfaces through a distributed communication protocol for robot autonomous control of a robot controller and a motor driver to collect real-time motor current data as well as joint torque, driving voltage, and real-time motion trajectory data in multiple dimensions.

[0012] In addition, the multidimensional feature vector extraction and preprocessing module comprises: a temperature data preprocessing unit that filters the slope of fluctuation in ambient temperature data that rises gradually due to the summer temperature rise of the factory itself or the deterioration of the ventilation fan in the distribution panel enclosure, and derives a temperature feature vector by extracting real-time deviation values ​​between the highest temperature components that occur rapidly in specific local areas among a plurality of grid regions detected by an infrared array temperature sensor; and a current high-frequency component separation unit that is linked and synchronized in real-time with the accurate detection time when the ultrasonic arc sensor captures micro-arc acoustics, removes commercial frequency and physical vibration fluctuation noise components due to the normal rotation of the motor or the movement inertia of the traveling bogie, and statistically separates and extracts only the transient frequency arc fingerprint components caused by insulation breakdown or equipment collision.

[0013] In addition, the machine learning-based intelligent anomaly detection engine is characterized by including: a probability density-based machine learning model that has been pre-learned from a normal data distribution accumulated when the process is in a normal state; and a real-time likelihood-related reliability value calculation unit that calculates a real-time likelihood-related reliability value, which is the probability that a deviation-based temperature feature vector, arc fingerprint component, joint torque, driving voltage, behavioral trajectory variation value, gas variation amount, and flame wavelength diagram will settle on the normal distribution.

[0014] In addition, the autonomous safety control and output module is characterized by including: an autonomous control execution unit that outputs a stop command by modifying the acceleration gradient in real time into a safe deceleration curve form, while simultaneously cutting off the main physical power supply at the hardware physical circuit level and triggering the fire extinguishing system to release a fire extinguishing agent; and a manager HMI visualization unit that ensures the safety of workers by displaying in real time on an edge touch display detailed process location information where the accident occurred, information on the cause of the anomaly, and evacuation guide information that guides the shortest emergency evacuation route for field workers.

[0015] Additionally, the control unit is characterized by performing the following steps: measuring ambient environment and fire sign data and collecting multidimensional robot dynamic feedback data in real time (S710); normalizing heterogeneous time series data and extracting micro-arc high-frequency feature components based on the combination of ultrasonic current time series (S730); inputting the extracted multidimensional feature vector into an artificial intelligence model to infer a real-time reliability value (S740); comparing the inferred real-time reliability value with an adaptive threshold (S750); if the reliability value is above the adaptive threshold as a result of the comparison, recognizing it as a simple load, driving inertia, or normal temperature fluctuation and driving the process continuously, and if it falls below the threshold, finally confirming it as an actual emergency accident situation (S770); and an autonomous safety control and output step (S800) that executes redundant safety control combining software and hardware at an ultra-low latency speed without intervention from an external server.

[0016] Additionally, the control unit comprises: a real-time data collection step (S710) for collecting data on ambient temperature, specific area IR array, ultrasound, gas, and flame from a composite sensor module, and acquiring heterogeneous field physical and behavioral data from a robot data acquisition module; a time series synchronization and fusion preprocessing step (S730) for extracting ambient temperature and specific area maximum temperature deviations from a temperature data preprocessing unit, and extracting ultrasonic arc sensing timing linkage, commercial noise filtering within motor current, and micro arc high-frequency feature components from a current high-frequency component separation unit; an AI-based inference step (S740) for applying a Gaussian mixture model normal distribution from an intelligent anomaly detection engine and calculating and inferring a real-time reliability value; an adaptive threshold comparison calculation step (S750) for comparing the calculated reliability value and the adaptive threshold in real-time; and a false alarm blocking control execution step (S760) for determining that if the reliability value is greater than or equal to the adaptive threshold, it is due to simple load, driving inertia, or normal temperature fluctuation, suppressing the safety control trigger, and continuing process operation. The method is characterized by performing the following steps: a step of finally determining the actual emergency accident situation when the threshold is lower (S770); a step of executing autonomous redundancy safety control without external control server intervention (S870); software safety control that generates a soft stop command communication message by modifying the acceleration slope safety deceleration curve in real time; autonomous safety control of hardware relay control of hardware relay cutoff driving and main physical power supply sudden cutoff, and ultra-low latency parallel batch execution of the output module (S800); and a step of simultaneously displaying the accident detailed process location, information on the cause of the anomaly, and information on the shortest emergency evacuation guide for field workers in real time (S810). Effects of the invention

[0017] This invention normalizes the deviation in real-time between the gradual rise in ambient temperature caused by seasonal fluctuations or ventilation facility conditions and the rapid rise in localized hot points (IR arrays), such as specific circuit breaker areas, thereby completely eliminating the risk of equipment malfunction and process stoppage caused by external thermal environment noise. Furthermore, it prevents production stoppage losses by perfectly distinguishing 'simple driving / motion loads' within the normal range—not only for floor-mounted robots but also for mobile robots with complex drive types involving AMR / AGV driving inertia or rail transport vibration—through the analysis of behavioral trajectory profiles and the verification of combined spatial density of environmental data. Additionally, by integrating heterogeneous multimodal sensors, such as ultrasonic arc, gas, and flame sensors, along with dual temperature feature vectors using Gaussian Mixture Model (GMM)-based spatial density, it can detect fire precursors at the very early stages using a precise sensor fusion method compared to a single sensor. Moreover, it processes inference and the issuance of control commands in a one-stop manner within the HMI edge device deployed on-site, without relying on the backbone network of an external cloud or central control server. Finally, upon confirmation of an emergency, it enables safe deceleration and stopping through the modification of the robot's distributed communication messages and physical hardware The relay power cutoff circuit operates simultaneously, maximizing safety. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram showing the configuration of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention. FIG. 2 is a block diagram showing the detailed configuration of the on-device HMI edge device of the present invention. FIG. 3 is a flowchart showing the operation method of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention. FIG. 4 is a detailed flowchart showing the operation method of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention. Specific details for implementing the invention

[0019] Hereinafter, with reference to the drawings, an on-device HMI edge device system having an integrated function for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI according to an embodiment of the present invention, and a method for operating the same, will be described in detail. In the following description, descriptions of conventionally known matters are omitted or simplified to clarify the gist of the present invention. The configurations included in the description of the present invention operate individually or in a composite combination.

[0020] FIG. 1 is a block diagram showing the configuration of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention.

[0021] Referring to FIG. 1, the on-device HMI edge device system includes: a data collection and linkage module (100) that measures surrounding environment and fire sign data and acquires multidimensional robot dynamic feedback data; a multidimensional feature vector extraction and preprocessing module (200) that normalizes heterogeneous time series data collected from the data collection and linkage module (100) and refines multidimensional feature vectors by matching timestamps based on a specific repetitive process cycle or a specific point-to-point movement drive cycle of a multi-joint robot; a machine learning-based intelligent anomaly detection engine (300) that calculates a real-time likelihood-related reliability value of a multidimensional feature vector output from the multidimensional feature vector extraction and preprocessing module (200) to determine a temporary simple load state and an actual accident state; and a control unit (5) that performs an autonomous safety control and output module (400) that controls the operation of a multi-joint robot and visualizes emergency information when the accident state is finally confirmed by the machine learning-based intelligent anomaly detection engine (300).

[0022] The multi-joint robot (10) is by no means limited to a fixed multi-joint robot arm that is attached to the floor of a manufacturing site and operated in a stationary state. The multi-joint robot (10) of the present invention includes all types of industrial or service robots capable of wired or wireless output of real-time driving feedback data (joint torque, motor current, driving voltage, and motion trajectory profile) during operation to an external communication line, such as a mobile multi-joint robot that is mounted on the upper part of a driving trolley structure such as an autonomous mobile robot (AMR) or an unmanned transport vehicle (AGV) and moves freely in a multidimensional space to perform a mobile manipulation process, a rail-driven robot in the form of a gantry that performs linear movement and high-speed acceleration and deceleration driving along a linear driving guide rail attached to the ceiling or wall of a factory, or a collaborative robot (Cobot) that shares physical space with a field worker to perform cooperative work.

[0023] The data collection and linkage module (100) detects the ambient temperature using a thermal imaging camera connected to an edge device when the robot moves and picks up a part, and the robot joint motor outputs current value (A) and torque data (Nm) in real time via a wired line.

[0024] The multidimensional feature vector extraction and preprocessing module (200) takes a specific process cycle of ‘3 seconds’ during which the robot picks up parts as a reference, and aligns the environment sensor data received 10 times per second and the robot current data received 100 times per second on the same timestamp, normalizes them (values ​​between 0 and 1), and combines them into one multidimensional feature vector.

[0025] When a robot temporarily lifts a heavy part and the current signal surges, the machine learning-based intelligent anomaly detection engine (300) calculates the likelihood reliability and determines it as a 'simple load' because there is no change in ambient temperature and the current rises temporarily, thereby preventing false alarms. On the other hand, if signs of fire (sudden temperature rise and abnormal torque drop) caused by a short circuit in the motor's internal wiring overlap, the likelihood reliability decreases rapidly and confirms it as an actual 'accident state'.

[0026] As soon as the autonomous safety control and output module (400) confirms the accident condition, the system issues an emergency stop (EMS) command to the AMR and robot arm to stop operation, and visualizes the location of the fire with a red warning light on the edge device screen (HMI).

[0027] The driving speed of the robot may slightly increase or decrease depending on the field workload or the weight of the parts, and if data is aligned only along a fixed time axis, the timing may be misaligned even during normal operation, leading to a misidentification of an abnormal state. The process cycle variable-responsive dynamic time warping alignment module incorporates a 'Dynamic Time Warping algorithm' within the multidimensional feature vector extraction and preprocessing module (200), and even if the robot's movement speed slows down by 10%, it recognizes the behavioral feature pattern of the process's 'start-progress-end' and flexibly extends or shortens the time axis of the time series data to match the timestamps.

[0028] Since unconditionally stopping the robot when an accident condition is finally confirmed (EMG Stop) paralyzes the entire line and causes economic losses, the 2-stage interlock and path bypass control module of the autonomous drive control adds a configuration that intelligently controls the accident by classifying the risk level of the accident. The autonomous safety control and output module (400) is subdivided into 'soft interlock' and 'hard interlock' stages, and in the 1st stage (soft interlock), if the likelihood of joint overheating increases, the driving speed of the robot is forcibly reduced to 50% or the real-time trajectory is redesigned so that the driving path of the mobile robot (AMR) bypasses the fire indication zone by more than 3 meters. In the 2nd stage (hard interlock), if the actual fire likelihood exceeds 95%, the main power is immediately cut off and the brake is applied.

[0029] Since the mobile robot (AMR) is powered by a battery, the battery will be discharged prematurely if the edge device constantly performs high-performance artificial intelligence computation and high-speed data collection. Therefore, the Eco Mode-based variable data sampling rate control module saves the battery by lowering the robot feedback data collection cycle to 50Hz and limiting the AI ​​inference cycle to once per second when the data collection module (100) is in normal conditions (when the likelihood of a normal state is high). Then, if even a slight likelihood of an abnormal sign, such as minute torque fluctuations or a rise in temperature, is detected, the collection cycle is immediately raised to 500Hz and the AI ​​engine is switched to a real-time ultra-high-speed inference mode (Boost mode).

[0030] FIG. 2 is a block diagram showing the detailed configuration of the on-device HMI edge device of the present invention.

[0031] Referring to FIG. 2, the data collection and linkage module (100) includes an ambient temperature sensor for measuring the air temperature and ambient temperature inside the distribution panel, a composite sensor module (110) for non-contactually scanning specific hazardous areas of circuit breakers or terminal blocks where loads are concentrated inside the distribution panel to generate grid-type area temperature data, measuring arc discharge shock wave acoustics, measuring off-gas, and measuring the intrinsic wavelength band of a flame; A robot data acquisition module (120) interfaced via a distributed communication protocol for autonomous robot control of a robot controller and a motor driver to collect real-time motor current data as well as joint torque, driving voltage, and real-time motion trajectory data in multiple dimensions; and a multidimensional feature vector extraction and preprocessing module (200) includes a temperature data preprocessing unit (210) that filters the slope of fluctuation in ambient temperature data that rises gradually due to the summer temperature rise of the factory itself or the deterioration of the ventilation fan in the distribution board enclosure, and extracts real-time deviation values ​​between the highest temperature components that occur rapidly in specific local areas among a plurality of grid regions detected by an infrared array temperature sensor to derive a temperature feature vector; A current high-frequency component separation unit (230) that is linked and synchronized in real-time with the precise detection time when an ultrasonic arc sensor captures micro-arc acoustics to remove commercial frequency and physical vibration fluctuation noise components due to the normal rotation of the motor or the movement inertia of the traveling bogie, and statistically separates and extracts only transient frequency arc fingerprint components caused by insulation breakdown or equipment collision; and a machine learning-based intelligent anomaly detection engine (300) includes a probability density-based machine learning model (310) that has been pre-learned from the normal data distribution accumulated when the process is in a normal state; and a real-time likelihood-related reliability value calculation unit (320) that calculates a real-time likelihood-related reliability value, which is the probability that a deviation-based temperature feature vector, arc fingerprint component, joint torque, driving voltage, behavioral trajectory fluctuation value, gas fluctuation amount, and flame wavelength map will settle on the normal distribution.The autonomous safety control and output module (400) includes an autonomous control execution unit (410) that outputs a stop command by modifying the acceleration gradient in real time into a safe deceleration curve shape, while simultaneously cutting off the main physical power supply at the hardware physical circuit level and triggering the fire extinguishing system to release a fire extinguishing agent; and a manager HMI visualization unit (420) that ensures the safety of workers by displaying in real time on an edge touch display detailed process location information where an accident occurred, information on the cause of the anomaly, and evacuation guide information guiding the shortest emergency evacuation route for field workers.

[0032] In a normal state (eco / simple load condition), the internal temperature of the factory rises gradually to 35℃ in midsummer, and the internal temperature rises slowly as the distribution panel ventilation fan ages. At this time, the temperature data preprocessing unit (210) filters the slope of the overall ambient temperature fluctuation into normal environmental changes, and at the same time, when the robot arm lifts heavy materials, the motor current and joint torque spike instantaneously, but the probability density-based machine learning model (310) confirms that this phenomenon settles in a normal distribution area within the process cycle and maintains the likelihood reliability at 98%.

[0033] When abnormal signs occur (micro-arc and insulation breakdown), the screws of the terminal blocks inside the switchboard loosen, and a small amount of off-gas is generated along with a micro-arc discharge. At this time, the ultrasonic sensor of the composite sensor module (110) captures the sound, and the current high-frequency component separation unit (230) mathematically removes the mechanical vibrations and commercial frequency noise generated by the robot's movement, and separates only the 'insulation breakdown arc fingerprint component'. The real-time deviation value between the maximum temperature of the 'specific local area' where the screws are loosened and the ambient air temperature in the grid-type area temperature data begins to widen rapidly.

[0034] In real-time identification and autonomous emergency control, the real-time likelihood reliability calculated by the numerical computation unit (320) drops to 30% or less, confirming the actual accident state. Immediately, the autonomous control execution unit (410) operates to change the acceleration gradient to a 'safe deceleration curve' and safely stops the moving robot without impact. At the same time, the main power of the switchboard is cut off at the hardware physical circuit level and the fire extinguishing agent is immediately released, and the manager HMI visualization unit (420) immediately displays the shortest emergency evacuation route on the touch display to allow the worker to move away from the toxic gas, along with the cause "Arc and overheating occurred in the switchboard B-hole Terminal 3 block."

[0035] Strong mechanical vibrations and electrical noise generated when the robot accelerates rapidly or the driving trolley passes over unevenness on the factory floor cause severe interference with the arc acoustic sensor of the composite sensor module (110). The robot motion trajectory-linked arc / vibration variable noise removal unit adds a 'robot motion-based adaptive filter' within the current high-frequency component separation unit (230), receives real-time motion trajectories (acceleration, velocity vector, number of joint rotations) from the robot data acquisition module (120), and calculates a 'vibration / frequency noise profile that is inevitably bound to occur due to robot driving' using a real-time mathematical model. Then, by subtracting this calculated robot-derived noise from the entire signal captured by the sensor, it finds only the pure switchboard insulation breakdown arc fingerprint.

[0036] A rise in the overall temperature of the factory or a failure of the ventilation fan in the summer heats up the entire inside of the distribution panel, but this does not cause an immediate fire. On the other hand, only a specific terminal block becoming hot is an immediate sign of fire. The temperature feature vector derivation unit based on an environmental degradation-local overheating dual matrix is ​​equipped with a 'two-dimensional matrix analysis method of spatial grid deviation and temporal variation slope' in the temperature data preprocessing unit (210), and diagnoses the deterioration state of the enclosure ventilation fan through the overall grid average temperature and variation slope (Δ / Δ) of the infrared array sensor, while constantly calculating the spatial deviation between a specific maximum temperature area and a surrounding area, and combines these two axes (temporal slope, spatial deviation) into a single matrix to generate a feature vector that double-verifies the 'simple environmental degradation state' and the 'local electrical short circuit state'.

[0037] If a moving robot arm is stopped abruptly, it may cause a secondary collision with a distribution board or surrounding facilities due to inertia, or cause fatal damage to the robot's mechanism, and if the timing of the release of the fire extinguishing agent is misaligned, the fire extinguishing efficiency decreases. The autonomous control execution unit's inertia momentum-preserving safety deceleration curve and fire extinguishing interlock crossover controller calculates the current center of gravity, trajectory velocity, and operating mass of the end-effector of the robot when the autonomous control execution unit (410) issues an emergency stop command, generates a 'dynamic inertia momentum-preserving deceleration curve,' safely draws a deceleration slope within the maximum deceleration limit (G-force) where the robot does not overturn or slip due to the operating mass, and triggers the fire extinguishing system while cutting off the main power of the hardware physical circuit at the final point of complete stop.

[0038] FIG. 3 is a flowchart showing the operation method of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention.

[0039] Referring to FIG. 3, the control unit measures ambient environment and fire sign data and collects multidimensional robot dynamic feedback data in real time (S710); normalizes heterogeneous time series data and extracts micro-arc high-frequency feature components based on the combination of ultrasonic current time series (S730); inputs the extracted multidimensional feature vector into an artificial intelligence model to infer a real-time reliability value (S740); compares the inferred real-time reliability value with an adaptive threshold (S750); if the reliability value is above the adaptive threshold as a result of the comparison, it is recognized as a simple load, driving inertia, or normal temperature fluctuation and the process is continuously driven, and if it falls below the threshold, it is finally confirmed as an actual emergency accident situation (S770); and performs an autonomous safety control and output step (S800) that executes redundant safety control combining software and hardware at an ultra-low latency speed without intervention from an external server.

[0040] In step S710, while the robot is driving and assembling parts, the composite sensor module measures the internal temperature of the distribution board and the arc ultrasonic sound, and the robot controller transmits the motor current (A) and real-time trajectory data to the control unit at a 10ms interval via distributed communication.

[0041] In step S730, the control unit normalizes data with different units to a range between 0 and 1, and at this time, mechanical vibration noise generated when the robot stops driving or accelerates rapidly is introduced into the ultrasonic sensor. The control unit mathematically combines the ultrasonic acoustics and the robot current time series data to eliminate noise originating from robot driving and precisely extracts only the 'micro-arc high-frequency fingerprint component' caused by electrical insulation breakdown to create a multidimensional feature vector.

[0042] Step S740 feeds this multidimensional feature vector into an on-device NPU-based probability density machine learning model in real time. The model calculates whether the current input value corresponds to the normal process data distribution and outputs a 'real-time confidence value (e.g., 92%)'.

[0043] In step S750, the control unit calculates an 'adaptive threshold (e.g., a reference point for variation of 45%)' in real time, which reflects the current robot's workload and the slope of factory ambient temperature fluctuations rather than a fixed threshold, and compares it with the current reliability value (92%).

[0044] In step S770, even if the robot lifts heavy materials and the motor current temporarily spikes, if the reliability value remains above the adaptive threshold (45%) at 80%, the control unit recognizes this as 'simple driving inertia and load' and continues to drive the process without interruption. On the other hand, if a fine arc high-frequency component is detected due to an actual short circuit in the switchboard and the reliability drops to 30%, it falls below the threshold, so it is immediately confirmed as an 'actual emergency accident situation'.

[0045] The S800 stage executes ultra-low latency control using only internal edge resources immediately upon confirmation of an accident, bypassing external clouds or main servers. At the software level, a 'safe deceleration curve command' is issued to the robot controller to safely decelerate and stop the robot; simultaneously, at the hardware physical circuit level, the main relay of the distribution panel is cut off and fire extinguishing agents are released. An evacuation route is displayed on the HMI screen.

[0046] If a fixed threshold is used, strong vibrations / currents generated when the robot is running at high speed (normal) are easily mistaken for fires / arcs (anomalies). The discrimination baseline (threshold) must be changed in real time according to the robot's driving state. The robot driving state pattern combined adaptive threshold generation unit includes a 'driving context-aware threshold generator' that receives real-time behavioral trajectories (acceleration, jerk, target torque) from the robot controller to perform the calculation of step S750. It dynamically controls the margin by temporarily lowering the discrimination threshold to prevent false alarms in sections where the robot accelerates rapidly or performs high-load processes, and strictly raising the threshold to detect even minute arcs in sections where the robot is stationary or running at constant speed.

[0047] If we look solely at ultrasonic acoustics, it responds to sounds of surrounding machinery colliding, and if we look solely at current high frequencies, it responds to motor noise. It is necessary to cross-verify whether these two heterogeneous time-series data occur 'simultaneously and in the same pattern.' The micro-arc fingerprint extraction unit based on ultrasonic acoustics-current high frequency time-series cross-correlation configures a 'cross-correlation coefficient calculator' in step S730 that receives ultrasonic arc detection time-series data from the composite sensor and current high-frequency fluctuation time-series data from the motor driver. It extracts a pure arc feature vector only when, during an actual electrical arc discharge, the ultrasonic shockwave acoustics in the air and the high-frequency spikes of the commercial current have a physical correlation within a specific microsecond (ms) error range, and the frequency entropy and cross-correlation coefficients of the two signals match a preset arc fingerprint profile.

[0048] In the event of an accident, if only the hardware physical power is cut off abruptly, the robot may collapse due to inertia, potentially leading to a major secondary accident; conversely, relying solely on software control prevents the power from being turned off in the event of a software down. The timing of these two types of control must be safely separated. The time-segmented crossover control unit of the software safety deceleration curve control and the hardware power physical cutoff interface is equipped with a 'time-segmented redundant safety sequencer' within the autonomous safety control and output module that performs the S800 stage. At the moment an accident is confirmed, the autonomous control execution unit immediately executes software control in the first stage by calculating the robot's current inertial mass, modifying the acceleration gradient, and inducing deceleration and evasive driving into a safe zone (ultra-low latency first-stage control). Subsequently, when it is confirmed that the robot's speed has reached a safe speed (e.g., 0.1 m / s or less) or a complete stop point, or as soon as the set safe dead time (e.g., 200 ms) elapses after the S / W control command, the main power supply of the hardware physical circuit is forcibly cut off in the second stage and the fire extinguishing agent is released (secondary physical control).

[0049] FIG. 4 is a detailed flowchart showing the operation method of an on-device HMI edge device system having a real-time fire prediction and robot anomaly detection integrated function based on multi-sensor fusion physical AI according to the present invention.

[0050] Referring to FIG. 4, the control unit collects data on ambient temperature, specific area IR array, ultrasound, gas, and flame from a composite sensor module and acquires heterogeneous field physical and behavioral data from a robot data acquisition module in a real-time data collection step (S710); extracts ambient temperature and specific area maximum temperature deviation from a temperature data preprocessing unit and extracts ultrasonic arc sensing timing linkage, commercial noise filtering within motor current, and micro-arc high-frequency feature components from a current high-frequency component separation unit in a time series synchronization and fusion preprocessing step (S730); applies a Gaussian mixture model normal distribution from an intelligent anomaly detection engine and calculates and infers a real-time reliability value in an AI machine learning model inference step (S740); compares the calculated reliability value with an adaptive threshold in real-time in an adaptive threshold comparison operation step (S750); and if the reliability value is greater than or equal to the adaptive threshold, determines it as simple load, driving inertia, or normal temperature fluctuation, suppresses the safety control trigger, and performs false alarm cutoff control to continuously drive the process (S760). If the threshold is not met, a step of finally determining the actual emergency accident situation (S770); a step of executing autonomous redundancy safety control without external control server intervention (S870); software safety control that generates a soft stop command communication message by modifying the acceleration slope safety deceleration curve in real time; a step of autonomous safety control of hardware relay control of hardware relay cutoff drive and main physical power supply sudden cutoff, and a step of executing the output module in ultra-low latency parallel batch (S800); and a step of simultaneously displaying the accident detailed process location, information on the cause of the anomaly, and information on the shortest emergency evacuation guide for field workers in real time (S810).

[0051] In step S710, while the robot is driving, a composite sensor module installed in the distribution board measures the ambient air temperature, IR array grid temperature, ultrasound, off-gas, and flame wavelength, and simultaneously acquires torque, voltage, and movement trajectory data from the robot controller.

[0052] In step S730, the control unit synchronizes heterogeneous data, during which strong electrical regenerative braking noise and rail friction vibrations generated when the robot rapidly decelerates are introduced. The current high-frequency component separation unit synchronizes with the ultrasonic sensing time to eliminate commercial noise and extract only the 'micro-arc high-frequency characteristic components' unique to insulation breakdown. The temperature data preprocessing unit derives the maximum temperature deviation of a specific terminal block amidst changes in ambient temperature.

[0053] Step S740 inputs the refined multidimensional feature vector into the on-device GMM (Gaussian Mixture Model) normal distribution to infer a real-time confidence value (e.g., 88%), which is the probability that the current process state belongs to the normal region, with ultra-low latency.

[0054] In steps S750 and S760, temporary torque load and vibration occur as the robot rapidly accelerates and transports heavy loads, and the control unit calculates an 'adaptive threshold (e.g., 50%)' reflecting the robot's current acceleration profile in real time and compares it with the current reliability (88%). Since it is above the threshold, the control unit determines (S760) that it is 'simple driving inertia and load' rather than a fire, suppresses the emergency trigger, and continuously drives the process without stopping.

[0055] Conversely, in steps S770 and S870, if a micro arc actually occurs due to a loose screw in the switchboard and the local temperature deviation increases rapidly, the reliability drops to 35% and falls below the adaptive threshold (50%). The control unit immediately confirms the actual emergency situation (S770) and executes autonomous redundancy safety control (S870) solely at the edge without network delay or intervention from the main control server.

[0056] In the S800 stage, as soon as an accident is confirmed, S / W and H / W controls are executed collectively as independent parallel threads. S / W safety control immediately modifies the robot's acceleration gradient to a 'safe deceleration curve' and transmits a soft stop message to prevent the robot from overturning on the rail. Simultaneously, H / W relay control abruptly cuts off the main power supply at the physical circuit level and releases fire extinguishing agents.

[0057] At the same time, S810 step protects the lives of workers by displaying the 'shortest emergency evacuation route' in real time on the edge touch display, along with cause information such as "Arc fire in terminal block 2 of switchboard area C," which allows evacuating to a safe zone while avoiding the direction of toxic gas spread.

[0058] If anomalies are detected simply by a fixed threshold, the high current and mechanical shock vibrations generated when the robot accelerates rapidly, travels at high speeds, or lifts heavy objects will be mistaken for switchboard arcs and fires. The robot's 'real-time driving state context' must be dynamically reflected inversely to the AI's judgment baseline. The robot motion trajectory profile-linked 'false alarm blocking (S760)' and 'adaptive threshold (S750)' generation unit is configured such that the control unit receives the real-time motion trajectory profile (target speed, instantaneous acceleration, jerk, joint torque) from the robot data acquisition module. Since the amount of vibration and current noise inflow increases in process sections where the robot performs violent acceleration and deceleration, the adaptive threshold variable calculator temporarily relaxes (downwardly adjusts) the judgment threshold baseline to maximize the false alarm blocking function of step S760. Conversely, in sections where the robot is stationary or driving at constant speed, the threshold baseline is strictly raised (upwardly adjusted) so that even minute arcs or gas leaks can be detected immediately.

[0059] If the ultrasonic sensor is used alone, it malfunctions due to the sound of surrounding tools colliding, and if only the high-frequency current is considered, it malfunctions due to motor inverter noise. Heterogeneous time-series data must be crossover-verified by utilizing the fact that the 'ultrasonic shock wave in the air' and the 'current spike within the wire,' which occur when electrical insulation breakdown (arc) takes place, are physically perfectly simultaneous. The high-frequency current time-series cross-correlation filtering unit based on the synchronization of the ultrasonic arc sensing time captures high-frequency components within the robot current data in a time window of microseconds (µs), based on the precise time when the composite sensor's ultrasonic sensor detects the micro-arc acoustic spike in step S730. Subsequently, the waveform entropy and cross-correlation coefficients of the two heterogeneous time-series signals are calculated; a pure arc feature vector is extracted only if it matches a predefined arc discharge unique fingerprint profile, while commercial frequency and rail vibration noise originating from the robot drive are mathematically completely canceled out and removed.

[0060] If only the physical hardware power is immediately cut off in the event of an accident, a gantry robot or robotic arm traveling at high speed may be propelled forward by inertia, crashing into a switchboard or derailing, resulting in a major secondary accident. Conversely, relying solely on software control prevents the power from being shut off when the main CPU goes down. These two types of control must be executed using 'scheduled parallel execution' tailored to the robot's physical state. In the S800 stage, the autonomous safety control and output module of the inertia momentum-preserving safety deceleration curve and the H / W physical cutoff interface operate the S / W control thread and H / W control circuit in parallel with a time delay; at the moment the accident is confirmed, the 'inertia momentum-preserving safety deceleration curve' is immediately executed via S / W trajectory control in Stage 1 based on the robot's current transport mass and speed, thereby rapidly decelerating the robot on the rail without slipping. Afterward, when the robot's operating speed reaches a safe limit speed (e.g., 0.1 m / s or less) without risk of overturning, the sensor confirms this, or immediately after the safety limit time (e.g., 150 ms) set in the hardware dead time timer elapses following the transmission of the S / W command, the H / W safety relay is triggered in the second stage to abruptly cut off the main power supply and release the fire extinguishing agent.

[0061] The present invention is not limited to the specific preferred embodiments described above, and any person skilled in the art to which the invention pertains can make various modifications without departing from the essence of the invention as claimed in the claims, and such modifications will be within the scope of the claims. Explanation of the symbols

[0062] 5: Control unit 10: Jointed robot 100: Data Collection and Integration Module 110: Composite sensor module 120: Robot Data Acquisition Module 200: Multidimensional Feature Vector Extraction and Preprocessing Module 210: Temperature Data Preprocessing Unit 230: Current high-frequency component separation unit 300: Machine Learning-Based Intelligent Anomaly Detection Engine 310: Probability Density-Based Machine Learning Model 320: Real-time Likelihood-Related Confidence Value Calculation Unit 400: Autonomous Safety Control and Output Module 410: Autonomous Control Execution Unit 420: Administrator HMI Visualization Unit

Claims

Claim 1 A data collection and interlocking module that measures surrounding environment and fire sign data and acquires multidimensional robot dynamic feedback data; and a multidimensional feature vector extraction and preprocessing module that normalizes heterogeneous time-series data collected from the data collection and linkage module and refines multidimensional feature vectors by matching timestamps based on a specific repetitive process cycle or movement drive cycle between specific points of a multi-joint robot; a machine learning-based intelligent anomaly detection engine that calculates a real-time likelihood-related confidence value of the multidimensional feature vector output from the multidimensional feature vector extraction and preprocessing module to distinguish between a temporary simple load state and an actual accident state; and a control unit that performs an autonomous safety control and output module that controls the operation of the multi-joint robot and visualizes emergency information when the accident state is finally confirmed by the machine learning-based intelligent anomaly detection engine; wherein the multidimensional feature vector extraction and preprocessing module includes a temperature data preprocessing unit that filters the slope of fluctuation in ambient temperature data that rises gradually due to the summer temperature rise of the factory itself or the deterioration of the ventilation fan of the switchboard enclosure, and derives a temperature feature vector by extracting real-time deviation values ​​between the highest temperature components that occur rapidly in specific local areas among a plurality of grid regions detected by an infrared array temperature sensor. An on-device HMI edge device system having integrated real-time fire prediction and robot anomaly detection functions based on multi-sensor fusion physical AI, characterized by including: a current high-frequency component separation unit that is linked and synchronized in real-time with the precise detection time when an ultrasonic arc sensor captures micro-arc acoustics, removes commercial frequency and physical vibration fluctuation noise components due to the normal rotation of a motor or the movement inertia of a traveling bogie, and statistically separates and extracts only transient frequency arc fingerprint components caused by insulation breakdown or equipment collision. Claim 2 The on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, characterized in that the data collection and interlocking module comprises: an ambient temperature sensor for measuring the air temperature and ambient temperature inside the distribution panel; a composite sensor module that non-contactually scans specific hazardous areas of circuit breakers or terminal blocks where loads are concentrated within the distribution panel to generate grid-type area temperature data, measures arc discharge shock wave acoustics, measures off-gas, and measures the intrinsic wavelength band of a flame; and a robot data acquisition module that interfaces via a distributed communication protocol for robot autonomous control of a robot controller and a motor driver to collect real-time motor current data as well as joint torque, driving voltage, and real-time motion trajectory data in multiple dimensions. Claim 3 delete Claim 4 The on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, wherein the machine learning-based intelligent anomaly detection engine comprises: a probability density-based machine learning model that has pre-learned a normal data distribution accumulated when the process is in a normal state; and a real-time likelihood-related reliability value calculation unit that calculates a real-time likelihood-related reliability value, which is the probability that a deviation-based temperature feature vector, arc fingerprint component, joint torque, driving voltage, behavioral trajectory variation value, gas variation amount, and flame wavelength diagram will settle on the normal distribution. Claim 5 The on-device HMI edge device system having integrated functions for real-time fire prediction and robot anomaly detection based on multi-sensor fusion physical AI, characterized in that the autonomous safety control and output module comprises: an autonomous control execution unit that outputs a stop command by modifying the acceleration gradient in real-time into a safe deceleration curve shape, simultaneously cuts off the main physical power supply at the hardware physical circuit level, and triggers a fire extinguishing system to release a fire extinguishing agent; and a manager HMI visualization unit that ensures worker safety by displaying in real-time detailed process location information where an accident occurred, information on the cause of the anomaly, and evacuation guide information guiding the shortest emergency evacuation route for field workers on an edge touch display.

Images