Vision Edge AI-based Process Safety and Production Integrated Management System
A vision-based system integrates safety and production management by using an image acquisition and edge computing to detect hazardous area intrusions and classify equipment states, reducing false alarms and duplicate counts, and enhancing data integration and response times in manufacturing environments.
Patent Information
- Application Number
- KR1020260077890
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-04-29
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-04-29
AI Technical Summary
Conventional safety management systems in manufacturing environments require separate sensor installations for monitoring hazardous zones and production performance, leading to installation burdens, maintenance challenges, and inaccurate alarms due to environmental factors like vibrations and temporary obstructions, while separate systems for safety and production management complicate integrated risk and production data utilization.
A vision-based system using an image acquisition unit and edge computing to detect worker intrusion into hazardous areas and calculate production counts, integrating safety and production management by classifying equipment states and adjusting intrusion criteria based on continuous frame analysis and multiple risk judgment criteria.
The system reduces false alarms and duplicate counts by using a single image system, provides real-time integrated management of safety and production data, and enhances maintenance convenience by eliminating the need for multiple sensors, thus improving operational data integration and response times in manufacturing environments.
Smart Images

Figure 112026052521142-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a safety management technology that monitors a work area of manufacturing equipment based on vision and determines whether a worker has entered a hazardous area using the monitoring results, and a production management technology that counts production performance based on the operational state transition of the equipment. More specifically, the present invention relates to a vision edge-based process safety and production integrated management system that utilizes an image acquisition device that photographs manufacturing equipment and an edge computing device that analyzes image frames acquired from the image acquisition device on-site to perform worker object detection, determination of hazardous area intrusion, determination of equipment state transition, and calculation of production counts, and to integrally relay, store, and visualize the results in a server and client environment.
[0002] The present invention is particularly applicable to manufacturing equipment in which repetitive up-and-down movements occur, such as press equipment, and relates to a technology that performs both hazardous zone monitoring and stroke-based production volume aggregation of said press equipment within a single image system. Furthermore, the detection of the worker object can be performed using a trained object detection model, and the detection result can be linked to subsequent safety judgments and production count processing in an edge computation environment. However, the scope of application of the present invention is not limited thereto and may also be applied to bending machines, reciprocating transfer devices, processing equipment, or other manufacturing equipment having repetitive machine operation states. Background Technology
[0003] Generally, manufacturing equipment such as press equipment performs high-speed repetitive motion, so there is a possibility of safety accidents such as pinching, collision, or impact if a worker's hand or part of their body approaches or enters a hazardous area. Accordingly, conventional manufacturing sites have used methods to monitor whether a worker approaches by installing separate safety devices such as optical sensors, proximity sensors, physical interlock devices, and emergency stop devices.
[0004] However, conventional safety management methods as described above require the configuration of separate sensor devices and wiring structures for each piece of equipment, which may result in a burden of installation and maintenance. Furthermore, if the equipment structure or work flow changes, reconfiguration or relocation may be required. Additionally, since these methods often make judgments based on whether a single condition is met or whether a sensor is simply blocked, rather than reflecting both the worker's actual entry movement and the equipment's operating cycle, unnecessary alarms may occur due to temporary obstruction, reflected light, shadows, or momentary shaking during work movements, or conversely, the response to actual dangerous situations may be delayed.
[0005] Meanwhile, methods relying on mechanical counters, contact switches, proximity sensors, or manual records by operators have been used for production performance management. However, these methods can lead to duplicate counting caused by vibrations or malfunctions unrelated to the actual state transitions of the equipment, and relying on manual aggregation may result in omissions or errors. Furthermore, if the safety monitoring system and the production performance aggregation system operate separately, it is difficult to comprehensively identify risk events and production history occurring within a single piece of equipment, which may limit the integrated utilization of on-site operational data.
[0006] Recently, attempts have been made to detect workers or assess equipment status using cameras and image processing technologies; however, determining entry into a hazardous zone based solely on the position of an object within a single frame, or counting production cycles based only on simple coordinate movements, makes it difficult to adequately reflect environmental characteristics of the manufacturing site, such as vibration, changes in illumination, object occlusion, reflected light, and the high-speed repetitive motion of equipment. Particularly in environments like press equipment, where the timing of equipment descent and the timing of a worker's hand entry and retrieval are closely linked, relying solely on simple object detection or position determination may limit the ability to interpret both actual risk and effective production status.
[0007] Therefore, there is a demand for technology that monitors the work zones of manufacturing facilities using a single vision system, more reliably determines whether a worker has entered a hazardous area, calculates production counts based on equipment state transitions, and enables the integrated management of the results in server and client environments. The problem to be solved
[0008] The present invention has been devised to solve the aforementioned problems, and one objective of the present invention is to provide a process safety and production integrated management system capable of monitoring whether a worker enters a hazardous area and calculating the production performance of the equipment by utilizing a single image acquisition system that captures the work area of a manufacturing facility and an edge computing system capable of on-site analysis.
[0009] Another objective of the present invention is to provide a safety management structure capable of reducing unnecessary alarms caused by temporary detection shaking, object occlusion, shadows, or reflected light by comparing location information of a worker object with region of interest information, applying a valid event judgment based on continuous frames rather than an immediate judgment based on a single frame, and variably adjusting the intrusion confirmation criteria by reflecting multiple risk judgment criteria and image instability judgment criteria together.
[0010] Another objective of the present invention is to provide a production coefficient structure capable of reducing redundancy caused by vibration or temporary shaking by not increasing the number of production cycles solely through simple positional movement of the equipment, but by classifying the equipment state into an upward state, a downward state, and a bottom-reaching state, and sequentially verifying the optical sealing state, structural stability state, and repulsion state for the transition candidates from the downward state to the bottom-reaching state.
[0011] Another objective of the present invention is to enable the safety and production status of a manufacturing site to be understood together in a single system by relaying and storing the safety management results, production count results, and related event data in a server unit, and visualizing them as real-time video, event history, and production status in a client unit.
[0012] The problems of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below. means of solving the problem
[0013] According to an embodiment of the present invention for solving the above problem
[0014] The system comprises: an image acquisition unit, which is a hardware component that captures a work area of a manufacturing facility and acquires image frames; an edge computation unit, which is a hardware component that receives the image frames from the image acquisition unit and analyzes the image frames on-site to determine whether a worker object has entered a danger zone and calculates the production count of the manufacturing facility; an alert output unit, which is a hardware component that outputs an on-site alarm based on the intrusion event determination result of the edge computation unit; a server unit, which is a hardware component that receives image-related data, intrusion event data, and production count data from the edge computation unit, and relays and stores them; and a client unit, which is a hardware component that connects to the server unit and visually displays the on-site status, intrusion events, and production count of the manufacturing facility. The edge computation unit includes: an object detection module that detects the worker object within the image frames and calculates a bounding box corresponding to the worker object; and a safety management module that determines whether the worker object has entered a danger zone by comparing the bounding box with a preset region of interest, and generates an intrusion event if the state of intrusion into the danger zone is maintained for more than a number of consecutive reference frames. and a production count module that determines the operating state of the manufacturing equipment from the image frame and increases the production count based on the validity verification result of the state transition candidate of the manufacturing equipment; wherein the image acquisition unit includes a global shutter type image sensor to reduce shutter distortion caused by the repetitive movement of the manufacturing equipment and further includes an infrared irradiation function; wherein the image acquisition unit is installed such that the main body, danger zone, and worker movement path of the manufacturing equipment are included within a single shooting range, and the shooting range is used as a reference image area for setting the area between the region of interest, the equipment reference point, and the mold; and the object detection module, after adjusting the image frame to match the model input format,The safety management module detects the worker object corresponding to the person class using a learned object detection model and calculates the bounding box; the safety management module sets one of the bottom center point of the bounding box, the reference point closest to the region of interest, or a sub-region corresponding to the hand object as an intrusion determination reference point; determines frame by frame whether the intrusion determination reference point is included within the region of interest; and generates the intrusion event only when the continuous inclusion state is maintained for more than the number of reference frames by accumulating the frame-by-frame determination results; the safety management module sets the sub-region corresponding to the hand area among the bounding boxes as a tracking target; does not maintain the number of reference frames as a fixed single value, but sets a basic number of reference frames as an initial value, and then variably calculates the number of reference frames based on a high-risk judgment criterion and an image instability judgment criterion; the high-risk judgment criterion is a phase proximity indicating where within the descent section the point at which the tracking target first crosses the boundary line of the region of interest exists, based on the total number of frames of the descent section from the point at which the slide of the manufacturing equipment starts descending to the point at which it reaches the bottom in one equipment operation cycle; A boundary line intersection ratio representing the ratio of frames in which the tracking target overlaps with the boundary line of the region of interest by more than a preset minimum intersection length or a preset minimum number of intersection pixels during a preset multiple frame interval; a direction alignment degree representing the degree of alignment between the center point movement trajectory vector of the tracking target and a target vector directed from the center point of the tracking target toward the center point of the region of interest; and a return delay calculated as a representative value of the number of frames taken from the point in time when the tracking target enters the region of interest to the point in time when it completely exits the region of interest for a preset multiple equipment cycles; wherein the image instability judgment criterion comprises:The safety management module includes an occlusion transition frequency indicating the number of times the state of the intersection area between the tracking target and the bounding box corresponding to the slide of the manufacturing equipment switches between a non-overlapping state and an overlapping state between consecutive video frames, and the safety management module calculates the number of items exceeding a threshold by comparing the phase proximity, the boundary line intersection ratio, the direction alignment, and the return delay with corresponding first, second, third, and fourth thresholds, respectively; subtracts one frame from the basic reference frame count if there is one item exceeding the threshold, subtracts two frames from the basic reference frame count if there are two items exceeding the threshold, and subtracts up to three frames from the basic reference frame count if there are three or more items exceeding the threshold; if the occlusion transition frequency exceeds a fifth threshold, adds one or two frames to the subtracted reference frame count, and calculates the final reference frame count by correcting the reference frame count after the subtraction and addition to a preset lower and upper limit value, and the safety management The module repeatedly collects the phase proximity, the boundary line intersection ratio, the direction alignment, the return delay, and the occlusion transition frequency for a normal operator during the calibration period at the beginning of installation, the first threshold value for the phase proximity is set by adding an upper allowable margin to the average value of the normal operation data, the second threshold value for the boundary line intersection ratio is set based on the upper quantile value of the normal operation data, the third threshold value for the direction alignment is set based on the upper limit of the normal distribution capable of distinguishing linear entry toward the center of the region of interest, the fourth threshold value for the return delay is set by adding a delay allowable margin to the median value or upper quantile value of the normal operation data, and the fifth threshold value for the occlusion transition frequency is set based on the upper quantile value of the object overlap transition frequency observed in the normal image section.The safety management module first performs a subtraction operation based on the high-risk judgment criteria and then performs an addition operation based on the occlusion transition frequency; if video frame data required for the calculation of some derived variables is missing, it treats the condition of exceeding the threshold for the missing derived variables as not being satisfied; if more than a preset ratio of all derived variables is continuously missing, it switches to a maintenance operation mode that restores the reference frame count to the basic reference frame count or sets the reference frame count to a value close to the upper limit; and when switched to the maintenance operation mode, it provides a notification to the client unit requesting a video quality check or a camera status check. The production count module analyzes the position change, contour change, or reference area occupancy status of the manufacturing equipment in the video frame to classify the operating state of the manufacturing equipment into one of an upward state, a downward state, or a bottom-reaching state; identifies the point in time when the state of the manufacturing equipment transitions from the downward state to the bottom-reaching state as a state transition candidate; but does not increase the production count merely because the state transition candidate has been identified; and regarding the state transition candidate, during the mold closing process A first condition including whether the arm connection width between the upper mold and the lower mold has decreased to a preset first value or less, and whether the state in which the arm connection width has decreased to a preset first value or less has been maintained for a preset number of floating frames or more; a second condition including whether the edge pattern fixation rate of the mold boundary in the frame section where the first condition is satisfied is greater than or equal to a preset allowable lower limit, and whether the descent symmetry error calculated as the absolute value of the difference in vertical pixel descent amounts between the left reference point and the right reference point is less than or equal to a preset allowable range; and after the first condition and the second condition are satisfied,The method is characterized by sequentially determining a third condition including whether the fine rebound displacement calculated as the reverse pixel movement amount of the equipment reference point in a preset multiple frame section immediately after reaching the bottom is less than or equal to a preset allowable upper limit; determining that one valid production cycle has been completed and increasing the production count by 1 only when the first condition, the second condition, and the third condition are all satisfied; transitioning to a count hold state without performing subsequent operations for the second condition and the third condition if the first condition is not satisfied; maintaining the count hold state without performing subsequent operations for the third condition if the first condition is satisfied but the second condition is not satisfied; and classifying the state transition candidate as an invalid state transition and not increasing the production count if the first condition and the second condition are satisfied but the fine rebound displacement in the third condition exceeds the allowable upper limit. The production count module is further characterized by the following: after the production count is increased by 1, until it is confirmed that the manufacturing equipment transitions from the bottom reaching state to the upward state Limiting the increase in production count, and collecting data on the arm connection width, the number of hold frames, the edge pattern fixation rate, the descent symmetry error, and the fine rebound displacement for multiple equipment cycles in which normal striking is repeated during the initial calibration period; the first value for the arm connection width is set by adding an allowable margin to the upper limit of the arm connection width upon reaching the bottom observed in the normal striking section; the number of hold frames is set based on the minimum number of frames or lower quantile value in which the arm connection width is maintained at a state of being reduced to or below the first value in the normal striking section; and the allowable lower limit of the edge pattern fixation rate is set based on the lower quantile value of the edge pattern fixation rate observed in the normal striking section.The allowable range of the above-mentioned descent symmetry error is set by adding an allowable margin to the upper limit of the difference in left-right descent amount observed in the normal impact section, and the allowable upper limit of the above-mentioned fine rebound displacement is set by adding an allowable margin to the maximum value of the fine rebound displacement observed in the normal impact section; if the above-mentioned edge pattern fixation rate drops below the above-mentioned lower allowable limit or the above-mentioned fine rebound displacement exceeds the above-mentioned upper allowable limit due to external impact or camera shake, the above-mentioned state transition candidate is classified as an invalid state transition; even if the above-mentioned dark area connection width temporarily decreases to below the above-mentioned first value due to changes in lighting or reflected light, if the state in which the above-mentioned dark area connection width decreases to below the above-mentioned first value is not maintained for more than the above-mentioned number of flight frames, it is determined that the above-mentioned first condition is not satisfied; and if it is difficult to normally calculate some of the above-mentioned dark area connection width, the above-mentioned edge pattern fixation rate, the above-mentioned descent symmetry error, or the above-mentioned fine rebound displacement due to frame omission, lens contamination, sudden change in illumination, or temporary failure to acquire images, it is treated as not satisfying the corresponding condition, or It may be characterized by withholding the evaluation of state transition candidates, temporarily suspending the increase of the production count if it is impossible to calculate the main reference value in more than a preset number of consecutive frames, and providing a notification to the client unit requesting a camera status check or a region of interest reset.
[0015] delete
[0016] delete
[0017] delete
[0018] delete
[0019] delete
[0020] delete
[0021] delete
[0022] delete
[0023] delete
[0024] delete
[0025] delete
[0026] delete
[0027] delete
[0028] delete
[0029] delete
[0030] delete
[0031] delete
[0032] delete
[0033] delete
[0034] delete
[0035] delete
[0036] delete
[0037] delete
[0038] delete
[0039] delete
[0040] delete
[0041] delete
[0042] delete
[0043] delete
[0044] delete
[0045] delete
[0046] delete
[0047] delete
[0048] delete
[0049] delete
[0050] delete
[0051] delete
[0052] delete
[0053] delete
[0054] delete
[0055] delete
[0056] delete
[0057] delete
[0058] delete
[0059] delete
[0060] delete
[0061] delete
[0062] delete
[0063] delete
[0064] delete
[0065] delete
[0066] delete
[0067] delete
[0068] delete
[0069] delete
[0070] delete
[0071] delete
[0072] delete
[0073] delete
[0074] delete
[0075] delete
[0076] delete
[0077] delete
[0078] delete
[0079] delete
[0080] delete
[0081] delete
[0082] delete
[0083] delete
[0084] delete Effects of the invention
[0085] According to the present invention, since the determination of whether a worker has entered a hazardous area and the calculation of the equipment's production count can be performed together using a single image acquisition system and an edge computation system, it can help unify the data flow at the equipment level compared to cases where a safety management system and a production management system are established separately.
[0086] In addition, according to the present invention, the relationship between the location information of a worker object and the region of interest is interpreted based on continuous frames, and the intrusion confirmation criteria can be variably adjusted according to multiple risk judgment criteria and image instability judgment criteria. Therefore, compared to a method that applies a fixed single standard frame count, the response time can be advanced in dangerous situations, and false alarms can be reduced in situations where object occlusion or detection shaking occurs.
[0087] In addition, according to the present invention, the equipment state is classified into a plurality of operating states, and the production count can be increased after sequentially verifying the arm sealing state, structural stability state, and repulsion state for the candidate transition from the downward state to the bottom reaching state, so it can be utilized to reduce duplicate counts caused by simple position changes or external vibrations.
[0088] In addition, according to the present invention, intrusion event information and production count information calculated by the edge computing unit are relayed and stored in the server unit, and can be displayed together with real-time video, event history, and production status in the client unit, thereby helping a field manager or a remote manager to check the equipment status in a more consistent manner.
[0089] In addition, according to the present invention, in a repetitive motion facility environment in which a press facility is the main embodiment, vision-based safety monitoring and production performance aggregation can be performed together without installing multiple separate sensors, thereby contributing to the integrated management of operational information by facility and the improvement of maintenance convenience.
[0090] The effects according to the present invention are not limited to those exemplified above, and a wider variety of effects are included within the present invention. Brief explanation of the drawing
[0091] Figure 1 illustrates a flowchart between all components according to the present invention. Figure 2 illustrates the dynamic hysteresis threshold frame calculation logic according to the present invention. Figure 3 illustrates the state transition validity verification logic according to the present invention. Figure 4 illustrates the worker-equipment synchronization judgment logic according to the present invention. Figure 5 illustrates an adaptive alarm intensity control logic according to the present invention. Specific details for implementing the invention
[0092] Embodiments of the present invention will be described in detail below with reference to the attached drawings. The embodiments described in this specification may be examples to explain the technical concept of the present invention, and the technical scope of the present invention should not be interpreted as being limited to specific embodiments. Accordingly, the embodiments described in the attached drawings and detailed description should be understood as representative examples to aid in understanding the present invention, and various modifications and variations may be possible within the scope equivalent to the technical concept of the present invention.
[0093] In this specification, terms including ordinal numbers, such as first, second, etc., may be used to distinguish multiple components from one another, and said terms are not to be interpreted as implying the order, importance, or necessity of said components. Furthermore, in this specification, expressions such as “comprising,” “comprising,” or similar expressions mean only that a specific component, step, or data is included, and are not to be interpreted as excluding the possibility of adding other components, steps, or data.
[0094] In addition, when one component is described as being “connected” or “connected” to another component in this specification, this may include not only cases of direct connection or connection, but also cases of connection or connection through the intermediary of another component. On the other hand, when it is described as being “directly connected” or “directly connected,” it may mean a connection without an intermediate component.
[0095] Meanwhile, as used in this specification, “module” or “unit” may refer to a component that performs at least one function or operation, and said module or unit may be implemented by hardware, software, or a combination of hardware and software. Additionally, a plurality of modules or a plurality of units may be implemented integrally in a single processor or a single computing device, and a single module or a single unit may be implemented by separating it into a plurality of sub-functional blocks.
[0096] Furthermore, regarding matters concerning general hardware control, communication procedures, or drive control that are not directly related to the core judgment logic of the present invention, configurations according to the prior art may be used in this specification, and detailed descriptions thereof may be abbreviated to reduce redundancy. However, such abbreviation does not apply to the core safety judgment logic, production count logic, or data processing procedures directly linked thereto of the present invention, and the core configurations and operations are described in detail below.
[0097] A system for implementing the present invention may include an image acquisition unit (100) for capturing a manufacturing facility and a surrounding workspace, an edge computation unit (200) for analyzing an image frame acquired from the image acquisition unit (100) at the site, an alert output unit (300) for outputting an on-site alarm according to the judgment result of the edge computation unit (200), a server unit (400) for relaying and storing image-related data and event data received from the edge computation unit (200), and a client unit (500) for connecting to the server unit (400) to check the on-site status.
[0098] The above system can simultaneously perform a safety function, which determines whether an operator has entered a hazardous area using image data acquired from a single vision device, and a production management function, which counts the number of operations of manufacturing equipment. Accordingly, instead of independently establishing separate safety sensor systems and separate production counter systems, an integrated management structure can be formed centered on image-based data flow.
[0099] The operation of the above system may include the steps of: the image acquisition unit (100) acquiring an image of the manufacturing equipment and workspace; the edge calculation unit (200) analyzing the image to determine the worker object and the equipment status; the safety management module (220) determining whether there is an intrusion into a dangerous area and determining whether to trigger an alarm; the production count module (230) identifying equipment state transition candidates and calculating a production count through a validity verification procedure described later; the server unit (400) relaying and storing the determination result and related data; and the client unit (500) visualizing and providing the data.
[0100] The data handled by the above system may include at least image frame data, object location data, region of interest data, equipment status data, intrusion event data, production count data, and time information data. The image frame data may be acquired from the image acquisition unit (100), the object location data may be calculated by the object detection module (210), and the intrusion event data and the production count data may be generated in an internal module of the edge calculation unit (200).
[0101] In one embodiment, the system may utilize a single camera device for photographing the press equipment, and the edge calculation unit (200) may simultaneously calculate whether there is an intrusion into the danger zone of the press equipment and the number of stroke-based productions. In another embodiment, the system may also be applied to a bending machine, a processing machine, a reciprocating transfer device, or manufacturing equipment that performs repetitive up-and-down movements. However, the main embodiment may be understood under the premise of a press equipment environment.
[0102] The image acquisition unit (100) may be a hardware component that acquires image frames by photographing a manufacturing facility and a surrounding workspace of the manufacturing facility. The image acquisition unit (100) may include a global shutter type image sensor in consideration of an environment in which equipment moving repeatedly at high speed is photographed, and may further include an infrared irradiation function to mitigate image quality degradation in a low-light environment.
[0103] The image acquisition unit (100) can continuously acquire image frames according to a fixed period or a high-speed shooting period. Additionally, the image acquisition unit (100) can be installed so that the manufacturing equipment body, the danger zone, and the worker's movement path are included within a single field of view. Accordingly, the image acquisition unit (100) can provide an input image for simultaneously performing worker object detection and determining the equipment operation status.
[0104] The image acquisition unit (100) can reduce shutter distortion by performing global shutter-based shooting, and accordingly, can relatively stably secure changes in the contour of the press slide performing high-speed reciprocating motion and changes in the position of the worker object. In addition, the image acquisition unit (100) can acquire visible light images, contrast images, or infrared auxiliary lighting-based images, and the image frames can be transmitted to the edge calculation unit (200) in the form of continuous frames. The transmission method may be a wired interface, an internal bus, or an integrated hardware internal transmission method.
[0105] The above image acquisition unit (100) can set a reference field of view at the time of installation and can adjust the shooting range including the location of the danger zone, the movement range of the equipment, and the area accessible to workers based on the reference field of view. The result of adjusting the shooting range can be used as a reference image area for setting the region of interest, detecting worker objects, and determining equipment state transition candidates, as described later.
[0106] In one embodiment, the image acquisition unit (100) may be implemented as an edge camera device including a stereo camera or a plurality of mono cameras. In another embodiment, the image acquisition unit (100) may be implemented as a high-speed camera capable of capturing at a level of 120 frames per second. In yet another embodiment, the image acquisition unit (100) may include an infrared irradiation unit to compensate for low-light environments.
[0107] The edge computation unit (200) may be a hardware component that directly analyzes an image frame input from the image acquisition unit (100) at the site. The edge computation unit (200) may be implemented as an integral unit with the image acquisition unit (100) or configured to perform low-latency analysis by being physically positioned adjacent to the image acquisition unit (100). The edge computation unit (200) may include an object detection module (210), a safety management module (220), and a production counting module (230).
[0108] The edge operation unit (200) can perform preprocessing on an input image frame, including at least one of resolution adjustment, region of interest-based cropping, noise reduction, or frame alignment, and then detect a worker object through the object detection module (210). Subsequently, the safety management module (220) can determine whether there is an intrusion into a danger zone by comparing the detection result with a preset region of interest, and the production count module (230) can identify a candidate for state transition by analyzing the motion state of the equipment within the image frame.
[0109] The edge computation unit (200) may transmit the entire judgment result to the server side, or may selectively transmit only the necessary result information and related frame information to the server side. Accordingly, the amount of transmitted data can be adjusted according to the field network status or server processing load.
[0110] The edge calculation unit (200) can receive image frames from the image acquisition unit (100) and can receive inputs including area of interest coordinates indicating a danger zone during the initial installation or management setting phase, frame continuity judgment criteria values, equipment status judgment criteria values, and notification output condition values. The edge calculation unit (200) can generate frame-specific object location information, intrusion status information, equipment status information, state transition candidate information, and production count information by using the input data and the image frames together.
[0111] In one embodiment, the edge computation unit (200) may be implemented in a form mounted on an internal computation board of a vision camera. In another embodiment, the edge computation unit (200) may be separately configured as a small computation device and connected to the image acquisition unit (100) via a short-range interface. In yet another embodiment, the edge computation unit (200) may operate in either a full frame transmission mode or an event-centered transmission mode depending on the field network status.
[0112] The object detection module (210) may be a software component that detects a worker object within the image frame and calculates a bounding box corresponding to the worker object. The object detection module (210) may use a deep learning-based object detection model.
[0113] The object detection module (210) can adjust the input image frame to match the model input format, determine whether it is a worker object, and output a location area for the worker object in the form of a bounding box. The object detection module (210) can generate detection results for each frame and transmit the generated results to the safety management module (220).
[0114] In one embodiment, the object detection module (210) may be configured to prioritize the use of detection results for a person class. In another embodiment, the object detection module (210) may be extended to secondarily identify gloves, protective gear, or foreign objects around equipment in addition to workers.
[0115] The object detection module (210) can receive an image frame transmitted from the edge operation unit (200) as input data, and can retrieve a learned model file and class filter setting values from pre-stored setting data. Additionally, in the event of a frame omission or a change in lighting, the object detection module (210) can temporarily refer to the detection result of the previous frame or the nearest valid frame to mitigate the abrupt interruption of location information transmitted to the safety management module (220).
[0116] The data produced by the object detection module (210) may include at least object class information and object location information. In one embodiment, the object detection module (210) may use a detection model based on YOLO version 6. In another embodiment, the object detection module (210) may use a model that has been additionally trained or corrected using images of a press workshop collected after on-site installation. In this case, the additional training or correction may be performed to reflect the frequency of missing worker objects, the frequency of false detection, or the degradation of detection quality due to changes in the lighting environment.
[0117] The above safety management module (220) may be a software component that determines whether a worker has entered a danger zone by comparing object location information calculated by the object detection module (210) with pre-set interest area information. Rather than generating an alarm based solely on immediate determination in a single frame, the safety management module (220) may determine a valid intrusion event by reflecting the continuity of the object location information, the positional relationship with the interest area, and the reference frame control logic described later.
[0118] The safety management module (220) can set a reference point among the bounding boxes calculated by the object detection module (210). In one embodiment, the reference point may be the center point at the bottom of the bounding box. The safety management module (220) can determine, frame by frame, whether the reference point is included within a pre-set area of interest. Subsequently, the safety management module (220) can accumulate the results of the frame-by-frame determination and determine the intrusion as a valid intrusion event if the intrusion state is maintained for more than a number of consecutive reference frames.
[0119] The above safety management module (220) does not maintain the reference frame number as a fixed single value, but can variably adjust it by reflecting the calibration data at the beginning of installation, the risk judgment criteria of the current frame section, and the image instability judgment criteria. Accordingly, when the worker's intention to enter is relatively clear and the temporal proximity to the equipment risk section is judged to be high, the reference frame number can be controlled to be reduced, and when object overlap, temporary occlusion, or detection shaking is repeated, the reference frame number can be controlled to be extended.
[0120] The safety management module (220) can receive a bounding box from the object detection module (210) and receive area of interest coordinates corresponding to the danger zone during the initial installation setup procedure. The area of interest coordinates can be stored in the form of a polygonal coordinate set, rectangular coordinate values, or a selected area on a reference image. Additionally, the safety management module (220) can perform a continuous frame determination by referring together to the frame number, time information, and the intrusion status of the previous frame.
[0121] In one embodiment, the safety management module (220) may generate an intrusion event when three or more consecutive intrusions are detected. In another embodiment, the safety management module (220) may generate the intrusion event only when five or more consecutive intrusions are detected. In yet another embodiment, the safety management module (220) may set the initial value or upper and lower limit range of the reference frame count differently depending on the daytime process, nighttime process, or time period when worker movement frequency is high. The specific calculation procedure for the reference frame count may be described in more detail in Process 1 below.
[0122] The production count module (230) may be a software component that analyzes the operating state of the manufacturing equipment from the video frame and calculates a production count corresponding to one work cycle of the manufacturing equipment. The production count module (230) may use a state machine-based determination structure instead of simple motion detection, and may increase the final production count only when the validity verification procedure described below is further performed on the state transition candidate.
[0123] The production count module (230) can classify the movement state of the manufacturing equipment into at least an upward state, a downward state, and a bottom reaching state. The production count module (230) can determine the current state by referring to the change in position, change in contour, or occupancy state of the equipment in a continuous frame. The production count module (230) can identify a case where the equipment state transitions from a downward state to a bottom reaching state as a state transition candidate, and can increase the production count by 1 only if the state transition candidate satisfies the verification conditions described below.
[0124] The above production count module (230) can mitigate duplicate counting by allowing the next count when the return to the upward state is confirmed after the bottom reaching state. Accordingly, the phenomenon of multiple counts occurring during the same stroke progression can be reduced.
[0125] The production count module (230) can use a continuous frame provided by the image acquisition unit (100) as input. Additionally, the production count module (230) can store the reference position of the equipment, the lower threshold position, the state transition allowance condition, and the count reset condition as setting data. The production count module (230) can generate frame-specific state values, state transition history, state transition candidate information, cumulative production count, and count occurrence time data.
[0126] In one embodiment, the production count module (230) may identify the point at which the downward movement of the press slide is confirmed to have reached the bottom point as a state transition candidate. In another embodiment, the production count module (230) may limit additional counting until the upward return is confirmed after reaching the bottom point. In yet another embodiment, the production count module (230) may subdivide the number of states into three or more depending on the type of equipment. For example, it may be subdivided into a standby state, an upward state, a downward state, a bottom reached state, and a return completed state. The procedure for validating the state transition candidate may be described in more detail in Process 2 below.
[0127] The above notification output unit (300) may be a hardware component that provides a warning to the work site when it receives an intrusion event signal or preliminary alarm stage information from the safety management module (220). The above notification output unit (300) may include at least one of a buzzer, a speaker, a lamp, a warning light, a site display device, or an output means linked to the screen display of the client unit (500).
[0128] The above notification output unit (300) may output at least one of an acoustic warning, an optical warning, or a screen warning based on whether the intrusion event is confirmed, a preliminary alarm stage, warning level information, or warning duration information. Additionally, the above notification output unit (300) may variably control the volume stage, warning sound repetition cycle, flashing cycle, light intensity stage, screen highlighting level, or warning display duration according to the adaptive alarm intensity control logic described later.
[0129] The server unit (400) may be a hardware component that receives image-related data, intrusion event data, and production count data calculated from the edge computation unit (200), and relays and stores them to multiple user terminals. The server unit (400) may include a data relay module (410) and a data storage module (420).
[0130] The server unit (400) can receive data from the edge computation unit (200) periodically or at the time of an event occurrence. The server unit (400) can deliver the received data to a user per session and simultaneously perform a log storage procedure. Additionally, the server unit (400) can manage data for real-time monitoring and stored data for history lookup separately. For example, recent frame data can be stored in a memory-based buffer, and event logs can be stored in a database.
[0131] The server unit (400) can connect to the edge computing unit (200) via a wired or wireless network. The server unit (400) can receive video frames themselves, event summary information, count values, or status value data, and the data format may be structured message data, frame data, or event packet data. Additionally, the server unit (400) can separately receive or store user authentication information, equipment identification information, site classification information, and visualization setting information.
[0132] In addition to simple relay and storage functions, the server unit (400) can aggregate subsequent evaluation data, such as average values of confirmation delays, unreleased alarm ratios, or user response history, and provide them to the adaptive alarm intensity control logic described later. Accordingly, the server unit (400) can function as an intermediate management layer connecting the primary judgment results on the edge side and the response history on the user side.
[0133] In one embodiment, the server unit (400) may be implemented as a cloud server. In another embodiment, the server unit (400) may be implemented as a local server installed in the internal computer network of the factory. In yet another embodiment, the server unit (400) may simultaneously manage edge devices at multiple sites and aggregate the frequency of risk events and production history for each site.
[0134] The data relay module (410) may be a software component that transmits data received within the server unit (400) to a plurality of client units (500) in real time or at low latency.
[0135] The data relay module (410) can relay frame information, intrusion event information, and production count information transmitted from the edge computing unit (200) to a number of connected user terminals using a real-time bidirectional communication protocol. The data relay module (410) may vary the scope of information provided according to user authority. Additionally, the data relay module (410) can set data priority by prioritizing the transmission of the event when it occurs and periodically transmitting general status information.
[0136] The data relay module (410) can retrieve real-time input data received by the server unit (400) from a buffer or message queue. Additionally, the data relay module (410) can configure transmission data by referring to connected user session information, transmission target facility identifier, and screen refresh cycle information. If necessary, the data relay module (410) can be operated in a way that reduces visualization delay by classifying warning level information, preliminary alarm stage information, or production count change events as priority transmission targets.
[0137] In one embodiment, the data relay module (410) may utilize a WebSocket-based connection structure. In another embodiment, the data relay module (410) may use a message queue so that event information and production count information are reflected sequentially even when multiple users connect simultaneously. In yet another embodiment, the data relay module (410) may selectively transmit only the data of a specific facility selected by the user.
[0138] The data storage module (420) may be a software component that records intrusion events occurring in the safety management module (220) and changes in production counts calculated in the production count module (230).
[0139] The data storage module (420) can record the time at which an intrusion event occurred, the duration of the event, the identification value of the related equipment, the location information of the object at the time the event occurred, and the time at which the production count changed in the database. The data storage module (420) can also configure indexes by date, equipment, and event type for history lookup. In addition, the data storage module (420) can set different storage cycles. For example, the production count can store the accumulated value at regular intervals, and the intrusion event can be stored immediately upon occurrence.
[0140] The data storage module (420) can receive event data, count data, and time data received through the server unit (400). Additionally, the data storage module (420) can store additional information such as related frame identification values, user confirmation status, warning release status, or alarm level history. Accordingly, the data storage module (420) can form a data base that can be utilized not only for simple history storage but also for subsequent maintenance analysis, false alarm checks, and user response history analysis.
[0141] In one embodiment, the data storage module (420) may store a timestamp, facility number, and event type when an intrusion event occurs. In another embodiment, the data storage module (420) may store the cumulative count and the update time together whenever the production count value is updated. In yet another embodiment, the data storage module (420) may record a capture image or a related frame number corresponding to the recent event together.
[0142] The above client unit (500) may be a hardware component that connects to the above server unit (400) to allow a remote manager to check the site status and visually identify risk events and production status. The above client unit (500) may include a monitoring module (510).
[0143] The client unit (500) can display real-time video or frame data received from the server unit (400) on a screen. Additionally, the client unit (500) can output worker location information and a point of interest display superimposed on the video. The client unit (500) can display at least one of the number of entry into a danger zone, total entry time, time of the most recent event, and production count information in numerical or graph form, and can view a list of recently occurred intrusion events in chronological order or check the production trend for a specific period.
[0144] The client unit (500) can receive real-time data streams, event log data, and summary data for dashboard display from the server unit (400). Additionally, the client unit (500) can generate facility selection values, inquiry periods, and display item setting values based on user input and transmit them to the server unit (400). If necessary, the client unit (500) can provide response data that can be used for calculating the average value of confirmation delay or the ratio of unreleased alarms described later by transmitting alarm confirmation operations, warning release operations, or event review history back to the server unit (400).
[0145] In one embodiment, the client unit (500) may be implemented as a web browser-based terminal. In another embodiment, the client unit (500) may be implemented as a factory control room monitor, a manager's mobile terminal, or a tablet device. In yet another embodiment, the client unit (500) may provide a real-time monitoring screen, an event history inquiry screen, and a production status dashboard, each in the form of separate screens or tabs.
[0146] The monitoring module (510) above is executed in the client unit (500) and may be a software component that visually combines and displays real-time video and event information.
[0147] The monitoring module (510) can display at least one of an object bounding box, a region of interest boundary line, a current safety status, a warning indicator, and a production count indicator by overlaying them on the received image. Additionally, the monitoring module (510) can provide a warning pop-up, a color change, or a sound output button in conjunction with the event at the time of event occurrence. In addition to real-time monitoring functions, the monitoring module (510) can retrieve stored event logs and display a comparison of the trend of risk events occurring during a specific time period and changes in production performance.
[0148] The monitoring module (510) can receive real-time video frames, object location information, event information, and statistical data for a dashboard from the server unit (400). Additionally, the monitoring module (510) can use the enlarged display section, equipment of interest, and viewing period as setting data according to user input. The monitoring module (510) can apply different popup sizes, border colors, flashing frequencies, or screen highlighting levels according to the warning level, preliminary alarm stage, or alarm intensity stage as needed.
[0149] In one embodiment, the monitoring module (510) may display the number of times the danger zone is entered and the total entry time in the form of a dashboard. In another embodiment, the monitoring module (510) may display the cumulative production count as a bar graph or a time series graph. In yet another embodiment, the monitoring module (510) may highlight the relevant frame when an intrusion event occurs and output the time of occurrence of the event and equipment information together.
[0150] The safety management module (220) of the edge calculation unit (200) may not apply the reference frame number, i.e., the hysteresis threshold frame number, for finally determining the intrusion of a worker object into a danger zone as a fixed single value, but may adjust it variably according to the operating state of the manufacturing equipment, the entry pattern of the worker object, and the recognition stability of the image frame. The reference frame number may be a determination criterion indicating how many consecutive frames must be maintained for the state in which the worker object is included within the region of interest to be confirmed as a valid intrusion event.
[0151] The safety management module (220) may apply a high-risk judgment criterion reflecting a dangerous situation and an image instability judgment criterion reflecting the instability of image recognition separately in order to adjust the number of reference frames. The high-risk judgment criterion may include phase proximity, boundary line intersection ratio, direction alignment, and return delay, and the image instability judgment criterion may include occlusion transition frequency. The high-risk judgment criterion may be used to reflect a state in which the likelihood of the worker object actually entering the danger zone increases, and the image instability judgment criterion may be used to reflect the likelihood of a one-time false detection occurring due to object overlap, temporary occlusion, or detection shaking.
[0152] The safety management module (220) can control the number of reference frames by reducing them, determining that the possibility of intrusion by the worker object has increased when the values corresponding to the high-risk judgment criteria exceed the corresponding threshold. Conversely, if the occlusion switching frequency exceeds the corresponding threshold, it can control the number of reference frames by extending them, determining that the stability of object identification in the video frame has decreased. Accordingly, the safety management module (220) can operate to advance the time of alarm confirmation when danger signs are clear, and to reduce the phenomenon where a one-time detection result leads directly to an alarm when video instability is high.
[0153] The above process 1 may consist of a default value setting step, a high-risk judgment criterion calculation step, a number of high-risk condition fulfillment calculation step, a reference frame count subtraction step, an image instability judgment criterion calculation step, a reference frame count addition step, an upper and lower limit correction step, and a final intrusion event confirmation step. At this time, the safety management module (220) may first perform a subtraction operation based on the high-risk judgment criterion and subsequently perform an addition operation based on the image instability judgment criterion. This order of operations may be intended to first secure responsiveness to dangerous situations and then correct for the possibility of excessive alarm generation due to image noise.
[0154] The above phase proximity may be a value indicating where within the descent section the point at which the hand bounding box of the worker object first crosses the boundary line of the area of interest exists, based on the total number of frames from the point at which the slide of the manufacturing equipment starts descending to the point at which it reaches the bottom in the most recent equipment operation cycle. For example, the safety management module (220) may identify the slide descent start frame number, the bottom arrival frame number, and the hand bounding box first intersection frame number, and calculate the phase proximity such that the phase proximity increases as the first intersection frame approaches the point at which it reaches the bottom. When the phase proximity increases, the entry point of the worker object can be interpreted as being temporally close to the dangerous descent section of the manufacturing equipment.
[0155] The above boundary line intersection ratio can be calculated as the ratio of frames in which the hand bounding box of the worker object overlaps with the boundary line of the area of interest during a recent preset multiple frame interval. In this case, the overlap is not simply a 1-pixel contact, but is considered satisfied when the hand bounding box and the boundary line of the area of interest overlap by more than a preset minimum intersection length or a preset minimum number of intersection pixels. For example, the safety management module (220) can set the last 10 frames as an evaluation window, calculate the number of frames in which an overlap greater than the minimum intersection length occurs within the evaluation window, and then divide this by the total number of evaluation frames to calculate the boundary line intersection ratio. Accordingly, it is possible to distinguish between cases where the hand object simply passes around the boundary line and cases where it continuously forms an entry state near the boundary line.
[0156] The above direction alignment degree may be a value representing the degree of alignment between the trajectory vector of the center point of the hand bounding box moving in consecutive frames and the target vector pointing from the center point of the hand bounding box toward the center point of the region of interest. In one embodiment, the safety management module (220) may extract the coordinates of the center point of the hand bounding box in a plurality of consecutive frames and calculate a movement vector pointing from the center point of the previous frame toward the center point of the current frame. Subsequently, the safety management module (220) may calculate the degree of direction matching between the movement vector and the target vector, and may calculate the direction alignment degree as the degree of direction matching increases. When the direction alignment degree increases, it may be interpreted as the worker object entering toward the center of the region of interest in a relatively consistent direction.
[0157] The above return delay can be calculated by collecting the number of frames taken for a plurality of recently set equipment cycles, from the point when the hand bounding box of the worker object enters the area of interest until the point when it completely exits to an external state that no longer overlaps with the area of interest. The safety management module (220) can store the number of dwell frames for each of the plurality of equipment cycles and calculate the return delay by extracting a median, average, or upper quantile value as a representative value from the set of stored dwell frames. In one embodiment, the return delay can be calculated as a median value so as not to be excessively affected by a single temporary long dwell value. If the return delay increases, it can be interpreted as a state in which the time the worker object stays inside the area of interest or in an adjacent area becomes longer.
[0158] The above occlusion transition frequency can be calculated by accumulating the number of times the state of the intersection area between the hand bounding box and the slide bounding box of the manufacturing equipment transitions from a non-overlapping state to an overlapping state, or from an overlapping state to a non-overlapping state, between consecutive video frames. The safety management module (220) can count the number of transitions between a state where the intersection area ratio is 0 and a state where it exceeds 0 within a preset evaluation time window or a preset evaluation frame interval. If the above occlusion transition frequency increases, the hand object and the equipment object may repeatedly overlap or separate in the video frame, and the object identification may be interpreted as being in an unstable state.
[0159] The safety management module (220) may initialize the reference frame number to a basic reference frame number for every video frame or every preset analysis cycle. In one embodiment, the basic reference frame number may be set to 5 frames. However, the basic reference frame number is not limited to a fixed value and may be set within a range of 3 to 7 frames depending on the number of strokes per minute of the equipment, the camera shooting frame rate, the range of worker movement speed, and the alarm operation policy at the site. The basic reference frame number may subsequently be subtracted or added based on the high-risk judgment criteria and the video instability judgment criteria.
[0160] The safety management module (220) can compare the phase proximity, the boundary line intersection ratio, the direction alignment, and the return delay with corresponding first, second, third, and fourth thresholds, respectively. If the phase proximity exceeds the first threshold, the safety management module (220) may determine that there is a high temporal proximity between the equipment descent risk zone and the time of worker entry. If the boundary line intersection ratio exceeds the second threshold, the safety management module (220) may determine that the hand object forms a repetitive or continuous entry state near the boundary line of the area of interest. If the direction alignment exceeds the third threshold, the safety management module (220) may determine that the hand object moves toward the center of the area of interest. If the return delay exceeds the fourth threshold, the safety management module (220) may determine that the hand object has increased the recovery delay.
[0161] The safety management module (220) calculates the number of high-risk judgment criteria that exceed the first to fourth thresholds and can progressively reduce the number of basic standard frames according to the calculated number. In one embodiment, if there is one high-risk judgment criterion that exceeds the threshold, one frame is deducted from the number of basic standard frames; if there are two, two frames are deducted; and if there are three or more, up to three frames can be deducted. Accordingly, the safety management module (220) can apply different detection sensitivities by distinguishing between cases where a single risk sign occurs and cases where multiple risk signs occur simultaneously.
[0162] After the subtraction step based on the above high-risk judgment criteria is completed, the safety management module (220) may compare the occlusion switching frequency with a fifth threshold. If the occlusion switching frequency exceeds the fifth threshold, the safety management module (220) determines that there is instability in object identification in the video frame and may add one or two frames to the reference frame count after the subtraction step. In one embodiment, if the occlusion switching frequency exceeds the fifth threshold but is less than the sixth threshold, one frame may be added, and if the occlusion switching frequency is greater than or equal to the sixth threshold, two frames may be added. At this time, the sixth threshold may be set to a value greater than the fifth threshold.
[0163] In cases where the safety management module (220) requires both a subtraction operation based on the high-risk judgment criteria and an addition operation based on the occlusion switching frequency simultaneously, the subtraction operation may be performed first, followed by the addition operation. For example, in a situation where the phase proximity, the boundary line intersection ratio, and the direction alignment all exceed a corresponding threshold and require a 3-frame subtraction, if the occlusion switching frequency also exceeds the corresponding threshold, the safety management module (220) may first subtract 3 frames from the basic reference frame count and then add 1 frame or 2 frames again according to the occlusion switching frequency. Accordingly, the responsiveness to a situation where risk signs overlap is reflected first, while subsequently correcting the possibility of false alarms due to image instability.
[0164] The safety management module (220) can correct the final calculated reference frame number so that it is located between a preset lower limit and an upper limit after both the subtraction operation and the addition operation are completed. In one embodiment, the lower limit may be set to 1 frame, and the upper limit may be set to 7 frames. If the final calculated reference frame number is smaller than the lower limit, it may be corrected to the lower limit, and if the final calculated reference frame number is larger than the upper limit, it may be corrected to the upper limit. Accordingly, the phenomenon in which the reference frame number is excessively reduced, causing an alarm to occur due to unstable detection of a single frame, or conversely, is excessively extended, causing the confirmation of an alarm regarding an actual dangerous situation to be excessively delayed, can be reduced.
[0165] The safety management module (220) can determine a valid intrusion event using the final calculated reference frame number. Specifically, the safety management module (220) can determine whether the state in which the reference point or hand bounding box of the worker object is included within the area of interest is maintained continuously for more than the final calculated reference frame number. If the inclusion state is maintained for more than the final calculated reference frame number, the safety management module (220) can determine the intrusion state as a valid intrusion event and transmit an alarm output signal to the notification output unit (300). Conversely, if the inclusion state is resolved before reaching the final calculated reference frame number, the safety management module (220) can treat the intrusion state as a temporary access or one-time detection result and withhold the generation of a valid intrusion event.
[0166] The first to fifth threshold values are not limited to a single fixed value but can be set based on normal operation data collected during the initial calibration period of installation. In one embodiment, the safety management module (220) may repeatedly collect phase proximity, boundary line crossing ratio, direction alignment, return delay, and occlusion switching frequency for a normal operator during the initial calibration period. The first threshold value for phase proximity can be set by adding an upper allowable margin to the average value of the normal operation data, and the second threshold value for the boundary line crossing ratio can be set based on the upper quantile value of the normal operation data. The third threshold value for direction alignment can be set based on the upper limit of the normal distribution capable of distinguishing linear entry toward the center of the region of interest, and the fourth threshold value for return delay can be set by adding a delay allowable margin to the median value or upper quantile value of the normal operation data. The fifth threshold value for occlusion switching frequency can be set based on the upper quantile value of the object overlap switching frequency observed in the normal image section.
[0167] In one embodiment, in a press equipment environment where 30 frames are captured per second, the basic reference frame count may be set to 5 frames. When the average value of the phase proximity is calculated to be approximately 0.67 during the initial calibration period of a normal operator, the safety management module (220) may set the first threshold value to approximately 0.75 by reflecting the upper allowable margin. Additionally, the boundary line intersection ratio may be treated as exceeding the threshold value if an overlap of more than the minimum intersection length occurs in 5 or more of the last 10 frames, and the return delay may be treated as exceeding the threshold value if the number of hand recovery frames increases by more than a certain range compared to the normal operation period. The occlusion transition frequency may be treated as exceeding the threshold value if the transition of the overlap state between the hand object and the equipment object within the recent evaluation window is repeated more than a preset number of times.
[0168] Additionally, if it is difficult to calculate some derived variables normally due to communication abnormalities at the work site, missing frames, lens contamination, sudden changes in illumination, or temporary failure to acquire images, the safety management module (220) may treat the relevant derived variables as not meeting the threshold value. However, if more than a preset ratio of all derived variables is continuously missing, the reference frame count may be restored to the basic reference frame count, or the safety management module (220) may be controlled to switch to a maintenance operation mode. In the maintenance operation mode, the reference frame count may be set to a value close to the default value or the upper limit value, and a notification requesting an image quality check or camera status check may be provided to the client unit (500).
[0169] The production count module (230) of the edge calculation unit (200) can calculate the production count after verifying whether the state transition of the manufacturing equipment is a valid strike corresponding to actual production. The production count module (230) can not increase the production count based solely on whether the position coordinates of the manufacturing equipment change from an upward state to a downward state or to a bottom reaching state, but can sequentially determine whether the state transition candidate of the manufacturing equipment corresponds to actual mold closing and strike actions based on a plurality of conditions.
[0170] The production count module (230) can classify the operating state of the manufacturing equipment into one of an upward state, a downward state, or a bottom-reaching state by analyzing the position change, contour change, or reference area occupancy state of the manufacturing equipment in a continuous video frame. The production count module (230) can identify the point in time when the state of the manufacturing equipment transitions from a downward state to a bottom-reaching state as a state transition candidate. However, the production count is not increased merely because the state transition candidate has been identified, and the final production count is increased only after sequentially verifying the first condition, the second condition, and the third condition described below.
[0171] The production count module (230) may apply a multi-stage pass structure in which a first condition, a second condition, and a third condition are sequentially combined to the state transition candidate. The first condition may be a condition for verifying an optical contact state based on the arm connection width and the number of floating frames occurring during the mold closing process. The second condition may be a condition for verifying a structural stability state based on the edge pattern fixation rate and downward symmetry of the mold boundary. The third condition may be a condition for excluding ineffective state transitions corresponding to mechanical bounce or image shake based on the fine rebound displacement immediately after reaching the bottom.
[0172] If the first condition is not satisfied, the production count module (230) may classify the state transition candidate as a count pending state and not perform subsequent operations for the second condition and the third condition. Additionally, even if the first condition is satisfied but the second condition is not satisfied, the production count module (230) may omit subsequent operations for the third condition and maintain the state transition candidate in a count pending state. Accordingly, the production count module (230) may not calculate all image features every time, but may control the system to perform only the necessary subsequent operations depending on whether the preceding conditions are satisfied.
[0173] The first condition above may be a condition for identifying the dark area connection state that appears during the process of actually closing the space between the upper mold and the lower mold. During the impact process of the press equipment, as the upper mold and the lower mold come closer to each other, the empty space between the molds is reduced, and the width of the dark area corresponding to the empty space may be reduced. The production count module (230) can distinguish between a simple coordinate movement and a state transition corresponding to the actual closing of the mold by utilizing the reduction of the dark area connection width and whether the reduced state is maintained.
[0174] The production count module (230) can set a region of interest that includes the area between molds within the image frame and calculate the dark area connection width formed between the upper mold and the lower mold within the region of interest. The dark area connection width can be calculated as a pixel-unit distance value by binarizing the empty space between molds within the region of interest or by thresholding it based on brightness contrast. For example, the production count module (230) can identify a continuous pixel section within the region of interest where the brightness value is below a preset brightness threshold as a dark area, and calculate the horizontal or vertical width of the dark area as the dark area connection width.
[0175] The above production count module (230) can calculate the number of frames held for how many consecutive frames the state in which the dark part connection width is reduced to a preset first value or less is maintained. The number of frames held may be a value that reflects the temporal stability of the mold remaining in the lower end state. For example, if the state in which the dark part connection width is reduced to a preset first value or less occurs for only one frame and is immediately resolved, it may be a change in state due to lighting flickering or temporary image shaking, so the production count module (230) may not recognize this as a valid first condition. Conversely, if the state in which the dark part connection width is reduced is maintained for more than the preset number of frames held, it may be determined that a part of the first condition is satisfied.
[0176] The production count module (230) can determine whether the arm connection width has decreased to a preset first value or less, and whether the decreased state has been maintained for a preset number of flight frames or more. If both the arm connection width decrease condition and the flight frame maintenance condition are satisfied, the production count module (230) can determine that the first condition is satisfied. On the other hand, if the arm connection width has not decreased to a preset first value or less, or if the decreased state is less than the number of flight frames, the production count module (230) can determine that the first condition is not satisfied and may withhold increasing the production count for the state transition candidate.
[0177] The second condition above may be a condition for verifying whether the lowering and bottom holding state of the manufacturing equipment is structurally stable, targeting the frame section identified as having satisfied the first condition above. The second condition may include the edge pattern fixation rate of the mold boundary and the descent symmetry of the manufacturing equipment. The edge pattern fixation rate may indicate whether the contour of the mold boundary is maintained relatively stably in the bottom holding section, and the descent symmetry may indicate whether the left and right reference points of the manufacturing equipment descend in a balanced manner.
[0178] The production count module (230) can extract a contour map or an edge map of the mold boundary in a frame section identified as satisfying the first condition. The production count module (230) can calculate the ratio of frames in which the amount of shape change of the mold boundary remains within a preset allowable range by comparing the edge map of an adjacent frame or a previous reference frame with the edge map of the current frame as the edge pattern fixation rate. The higher the edge pattern fixation rate, the more the shape of the mold boundary can be interpreted as being maintained relatively stably in the lower maintenance section. Conversely, if the edge pattern fixation rate is low, the contour of the mold boundary can be interpreted as being in an unstable state due to camera shake, reflected light, image blur, or abnormal vibration of the equipment.
[0179] The production count module (230) can set a left reference point and a right reference point, respectively, on the slide or mold structure of the manufacturing equipment. The left reference point and the right reference point can be set as the coordinates of a mold corner, a slide outer point, or a reference marker that can be repeatedly tracked in the image frame. The production count module (230) can calculate the vertical pixel drop amount while the left reference point and the right reference point move from an upper position to a lower position, respectively, and calculate a drop symmetry error using the absolute value of the difference between the two drop amounts. The smaller the drop symmetry error, the more likely the manufacturing equipment is interpreted as having dropped while maintaining left-right balance; conversely, if the drop symmetry error increases, it may be interpreted as reflecting asymmetric drop, image distortion, or camera shake.
[0180] The production count module (230) can determine whether the edge pattern fixation rate is above a preset allowable lower limit and whether the descent symmetry error is below a preset allowable range. If both the edge pattern fixation rate condition and the descent symmetry condition are satisfied, the production count module (230) can determine that the second condition is satisfied. On the other hand, if the edge pattern fixation rate drops below the allowable lower limit or the descent symmetry error exceeds the allowable range, the production count module (230) does not recognize the state transition candidate as a normal bottom-reaching state and may withhold the increase in the production count.
[0181] The third condition above may be a condition to exclude mechanical bouncing, abnormal rebound, or temporary image shaking that may occur immediately after reaching the bottom after both the first condition and the second condition are satisfied. The production count module (230) can check whether the equipment reference point moves in the opposite direction to the original downward direction in a short frame interval immediately after the manufacturing equipment reaches the bottom state. If the reverse movement appears excessively, it may be determined that it corresponds to an external impact, camera shake, equipment bouncing, or abnormal state transition rather than a state where the actual impact has been stably completed.
[0182] The production count module (230) can calculate the amount of reverse pixel movement of the equipment reference point in a pre-set multiple frame interval for a second time point immediately after reaching the bottom. In one embodiment, the multiple frame interval may be set to 2 to 4 frames immediately after reaching the bottom. The production count module (230) can accumulate the amount of pixels in which the equipment reference point has moved in the direction opposite to the downward direction, or calculate the maximum amount of reverse movement observed in the multiple frame interval as a fine rebound displacement. The smaller the fine rebound displacement, the more the stationary state immediately after reaching the bottom can be interpreted as relatively stable, and if the fine rebound displacement increases, it can be interpreted as a state reflecting abnormal rebound or image shaking.
[0183] The production count module (230) can determine whether the fine rebound displacement is below a preset allowable upper limit. If the fine rebound displacement is below the allowable upper limit, the production count module (230) can determine that the third condition is satisfied. On the other hand, if the fine rebound displacement exceeds the allowable upper limit, the production count module (230) may classify the state transition candidate as an invalid state transition and not perform a production count increase.
[0184] The production count module (230) may determine that one valid production cycle has been completed only when the first condition, the second condition, and the third condition are all satisfied. When it is determined that the valid production cycle has been completed, the production count module (230) may increase the cumulative production count by 1 and transmit the time of the production count increase, the equipment identification information, and the state transition history to the server unit (400). Conversely, if any of the first to third conditions are not satisfied, the production count module (230) may not perform the production count increase and may maintain a count hold state.
[0185] The production count module (230) may not allow the next increase in the production count until it is confirmed that the manufacturing equipment has transitioned from the bottom state to the upward state or the return completion state after the production count has been increased by 1. Accordingly, even if multiple state transition candidates are detected in the bottom holding section of the same stroke, the production count module (230) can process this as a single production cycle. Additionally, a new state transition candidate in the next downward state can be evaluated only after the upward state or the return completion state has been confirmed.
[0186] The production count module (230) can collect data on arm connection width, number of airborne frames, edge pattern fixation rate, downward symmetry error, and fine rebound displacement for a plurality of equipment cycles in which normal strikes are repeated during the initial calibration period. The normal strike can be identified based on image frames acquired in a section designated as a reference strike section by a manager, a section where the product is confirmed to have been discharged normally, or a section where the equipment is set to a normal operation mode. The production count module (230) can set each reference value used for the first to third conditions based on the upper limit, lower limit, median, or percentile value of the normal strike data.
[0187] In one embodiment, the first value for the arm connection width may be set by adding an allowable margin to the upper limit of the arm connection width at bottom arrival observed in the normal impact section. The number of hang frames may be set based on the minimum number of frames or lower quantile value in which the reduced state of the arm connection width is maintained in the normal impact section. The allowable lower limit of the edge pattern fixation rate may be set based on the lower quantile value of the edge pattern fixation rate observed in the normal impact section. The allowable range of the descent symmetry error may be set by adding an allowable margin to the upper limit of the difference in left-right descent amount observed in the normal impact section. The allowable upper limit of the fine rebound displacement may be set by adding an allowable margin to the maximum value of the fine rebound displacement observed in the normal impact section.
[0188] In one embodiment, the fine rebound displacement measured immediately after reaching the bottom during 10 normal strikes of the press equipment may appear at levels of 8 pixels, 11 pixels, 7 pixels, 12 pixels, 10 pixels, 9 pixels, 13 pixels, 8 pixels, 10 pixels, and 11 pixels. The average rebound displacement for the sample may be calculated to be approximately 9.9 pixels, and the maximum value observed in the normal strike section may be summarized as 13 pixels. The production coefficient module (230) may set the allowable upper limit of the fine rebound displacement to 15 pixels by adding a preset allowable margin to the maximum value of the normal strike section. In this case, a state transition in which the fine rebound displacement immediately after reaching the bottom is 15 pixels or less may be treated as satisfying the third condition, and a state transition in which the fine rebound displacement exceeds 15 pixels may be classified as an invalid state transition.
[0189] In one embodiment, the first value for the dark area connection width may be set in the range of several pixels to tens of pixels depending on the shooting resolution, mold spacing, lighting conditions, and the contrast of the reference image. For example, when the dark area connection width is located in the range of 4 pixels to 6 pixels upon reaching the bottom in the normal impact section, the production count module (230) may set the first value to the level of 7 pixels by reflecting the allowable margin. Additionally, the allowable lower limit of the edge pattern fixation rate may be set based on the range in which the edge map matching rate of the mold boundary in the normal impact section is maintained at a certain level or higher. For example, when the edge pattern fixation rate in the normal impact section is located in the range of 88 percent to 95 percent, the allowable lower limit may be set to the level of 85 percent. The allowable range of the above-mentioned descent symmetry error can be set to a value of 10 pixels or around that value by reflecting an allowable margin when the difference in the amount of vertical descent between the left reference point and the right reference point in the normal impact section is located in the range of 5 pixels to 9 pixels.
[0190] The production count module (230) can determine that the first condition is satisfied if, in a normal impact scenario, the arm connection width decreases to less than the first value and the reduced state is maintained for more than the preset number of flight frames. Subsequently, in the frame section where the first condition is satisfied, the edge pattern fixation rate is maintained above the allowable lower limit and the descent symmetry error is below the allowable range, so the second condition is satisfied. Then, if the fine rebound displacement immediately after reaching the bottom is below the allowable upper limit, the third condition is satisfied. When all three conditions are satisfied, the production count module (230) can recognize the state transition candidate as a valid production cycle and increase the production count by 1.
[0191] When the camera is temporarily shaken due to external shock or floor vibration occurring around the manufacturing facility, the coordinates of the facility reference point within the video frame may be observed to have moved to a position corresponding to the bottom arrival state. However, the production count module (230) can check whether the edge pattern fixation rate in the second condition is lowered below the allowable lower limit, and whether the fine rebound displacement in the third condition exceeds the allowable upper limit. For example, if the amount of reverse pixel movement immediately after bottom arrival is observed to be at the level of 20 pixels due to camera shaking, the fine rebound displacement exceeds the allowable upper limit set at the level of 15 pixels, so the production count module (230) may classify the state transition candidate as an invalid state transition and not perform a production count increase.
[0192] If the workshop lighting flickers temporarily or reflected light occurs around the mold, the contrast within the area of interest changes instantaneously, and the width of the dark area connection between the molds may be calculated to be smaller than the actual width. Even in this case, the production count module (230) can check whether the state in which the width of the dark area connection is reduced to or less than the first value is maintained for more than a preset number of frames. If the state of reduced width of the dark area connection appears only in one frame or a short frame interval and is immediately resolved, the production count module (230) can determine that the first condition is not satisfied and may withhold increasing the production count. Additionally, even if the first condition is accidentally satisfied, if the edge pattern fixation rate or the descent symmetry error in the second condition exceeds the allowable range, the production count module (230) may classify the state transition candidate as an invalid state transition.
[0193] The production count module (230) may treat the conditions as not being met if it is difficult to normally calculate some of the dark area connection width, edge pattern fixation rate, falling symmetry error, or fine rebound displacement due to frame omission, lens contamination, sudden change in illumination, or temporary image acquisition failure. However, if the omission of the image data is temporary, the evaluation of the state transition candidate may be withheld by referring to the state value of the previous valid frame or the nearest valid frame. Additionally, if it is impossible to calculate the main reference value in a number of consecutive frames greater than a preset number, the production count module (230) may temporarily stop increasing the production count and provide a notification to the client unit (500) requesting a camera status check or a reset of the area of interest.
[0194] The production count module (230) can distinguish between the count increase result approved as a valid production cycle and the history of a state transition candidate that is suspended because it does not satisfy any of the first to third conditions, and transmit this to the server unit (400). The data storage module (420) of the server unit (400) can store the count increase time, the equipment identification information, the time of occurrence of the state transition candidate, the judgment result for each condition, and the related frame identification value. Accordingly, the administrator can check not only the production count history through the client unit (500) but also whether the cause of the count being suspended corresponds to the arm connection width condition, the edge pattern fixation rate condition, the downward symmetry condition, or the fine rebound displacement condition.
[0195] The safety management module (220) of the edge computation unit (200) is not limited to a method of outputting an alarm only after the worker object has actually entered the danger zone, but can provide a preliminary notification at a stage prior to the actual intrusion event by analyzing the temporal relationship between the worker behavior pattern accumulated over multiple equipment cycles and the operation pattern of the manufacturing equipment. Process 3 may be a procedure for classifying the worker-equipment synchronization state using the repetitive hesitation, dwelling, change of direction, dispersion of the insertion time, and reduction of the retrieval margin observed by the hand object of the worker object near the boundary line of the area of interest.
[0196] Process 3 above can be performed auxiliaryly before the confirmation of a valid intrusion event according to Process 1 above. That is, the safety management module (220) can control the output of a preliminary alarm when the hand object of the worker object shows an input / retrieval rhythm different from the usual near the boundary of the area of interest, even when the worker object has not yet confirmed entry into the area of interest. Accordingly, Process 3 does not replace the logic for confirming a valid intrusion event, but can function as an additional safety judgment procedure that detects changes in the work rhythm in the stage prior to the occurrence of the valid intrusion event.
[0197] The above safety management module (220) can calculate the number of hesitations, the dwell rate of the inner band area of the boundary line, the accumulated amount of direction change, the phase concentration, and the recovery margin sub-quantile value to determine the worker-equipment synchronization state. The number of hesitations, the dwell rate of the inner band area of the boundary line, and the accumulated amount of direction change may be criteria for determining local behavioral instability occurring near the boundary line of the area of interest. The phase concentration and the recovery margin sub-quantile value may be criteria for determining the temporal misalignment between the repetitive cycle of the manufacturing equipment and the time of input and retrieval of the worker hand object.
[0198] The safety management module (220) can set a plurality of recently set equipment cycles as a single evaluation window. In one embodiment, the evaluation window can be set to the last 10 equipment cycles, and in another embodiment, the evaluation window can be set to the last 30 seconds, 1 minute, or a pre-set time interval. The safety management module (220) can compare the judgment criteria calculated in the current evaluation window with the judgment criteria calculated in the previous evaluation window or the reference evaluation window. The reference evaluation window can be set from normal operation data of the initial calibration period and can be updated according to the accumulated normal operation history thereafter.
[0199] The safety management module (220) may use at least one of the following operating states of the manufacturing equipment as a reference point for the equipment cycle: a point of descent start, a point of reaching the bottom, a point of descent start, and a point of return completion. In one embodiment, the point of descent start may be identified as the first frame in which the production count module (230) classifies the state of the manufacturing equipment as a downward state, and the point of reaching the bottom may be identified as the first frame in which the manufacturing equipment is classified as a bottom-reaching state. The safety management module (220) may use the reference points to convert the point of input and the point of retrieval of a hand object into relative positions within one equipment cycle.
[0200] The safety management module (220) may set a sub-region corresponding to the hand area among the worker object bounding boxes calculated by the object detection module (210), or the bottom center point, side reference point, or object area presumed to be a hand of the worker object bounding box as a tracking target. If the hand object is detected as a separate class, the bounding box of the hand object may be used directly; if the hand object is not detected as a separate class, a part of the worker object bounding box or the reference point closest to the area of interest may be used as the representative coordinates of the hand object. The representative coordinates may be the center point coordinates per frame, the boundary line nearest point coordinates, or the bounding box corner coordinates protruding in the direction of the area of interest.
[0201] The above number of hesitations can be calculated by accumulating the number of times the direction of movement is reversed as the representative coordinates of the hand object repeat the entry and retreat directions within sub-sections set inside and outside the boundary line of the area of interest. The safety management module (220) calculates a movement vector using the representative coordinates of the hand object for each consecutive frame, and counts as a direction reversal event cases where the direction of the movement vector switches from the center direction of the area of interest to the outside direction of the area of interest, or conversely, switches from the outside direction of the area of interest to the center direction of the area of interest. However, the direction reversal event can be counted only when it involves a pre-set minimum movement distance or greater. Accordingly, it is possible to reduce the excessive reflection of single-pixel level shaking or detection noise in the number of hesitations.
[0202] The dwell rate of the inner band area of the boundary line can be calculated by converting the number of frames in which the hand object stays within a band area formed with a predetermined width immediately inside the boundary line of the area of interest into a ratio to the total number of evaluated frames. The band area can be defined as a section extended inward by a certain pixel width from the boundary line of the area of interest, and the width can be set considering the camera resolution, the size of the area of interest, and the actual size of the facility risk area. The safety management module (220) can count frames in which the representative coordinates of the hand object or a part of the hand object bounding box overlap with the band area as dwell frames. If the dwell rate of the inner band area of the boundary line increases, it can be interpreted that the tendency for the hand object to stay in the danger-adjacent section has increased even if it does not enter deep into the area of interest.
[0203] The above-mentioned accumulated amount of direction change can be calculated by accumulating and summing the absolute values of the rotation angles between the movement trajectory vectors formed by connecting the representative coordinates for each consecutive frame of the hand object. The safety management module (220) calculates the amount of change in direction between the previous movement vector and the current movement vector, and can reflect the amount of change in direction in the accumulated amount of direction change only if the amount of change in direction is greater than or equal to a preset minimum angle. Additionally, if the movement distance of the hand object is less than the preset minimum movement distance, it can be treated as a stationary state or a detected shake and excluded from the calculation of the accumulated amount of direction change. If the accumulated amount of direction change increases, the trajectory of the hand object may be interpreted as a state in which it fails to maintain a simple insertion and retrieval pattern and shakes repeatedly near the boundary line.
[0204] The above phase concentration can be calculated as a value indicating how the input start times of the hand object are clustered in a certain interval across multiple equipment cycles, after normalizing the input start time of the hand object by corresponding it to the standard cycle of the manufacturing equipment. In one embodiment, the safety management module (220) defines the period from the descent start time of each equipment cycle to the next descent start time as one cycle, and can identify the frame in which the hand object first approaches the boundary line of the area of interest or the sub-section outside the boundary line as the input start time. Subsequently, the safety management module (220) can convert the input start time into a period position value or circular coordinate value between 0 and 1, and calculate the phase concentration by evaluating the dispersion of the converted values as a circular statistical amount. The larger the phase concentration value, the more the input start time of the hand object may be relatively concentrated in a certain interval within the equipment cycle, and the smaller the phase concentration value, the more the input start time may be dispersed throughout the equipment cycle.
[0205] The above recovery margin lower percentile value can be calculated by collecting the number of margin frames secured from the point when the hand object completely deviates from the region of interest or the adjacent section of the region of interest in a recent plurality of equipment cycles until the point when the slide of the manufacturing equipment starts to descend again, and then extracting a representative value corresponding to the lower percentile section among the collected number of margin frames. In one embodiment, the recovery margin lower percentile value can be calculated as a representative value of the lower 25 percent section. If the recovery margin lower percentile value decreases, it can be interpreted as a state in which the minimum margin time secured from the recovery of the hand object until the next equipment descent decreases. Accordingly, the safety management module (220) can separately detect a situation in which the recovery margin decreases rapidly in some cycles, even if the average recovery margin is sufficient.
[0206] The safety management module (220) can determine a local instability state by comparing the number of hesitations, the dwell rate of the inner band area of the boundary line, and the accumulated amount of direction change calculated in the current evaluation window with the values of the previous evaluation window or the reference evaluation window. In one embodiment, if the number of hesitations increases by more than a preset increase rate compared to the average value of the reference evaluation window, or the dwell rate of the inner band area of the boundary line increases by more than a preset increase rate compared to the dwell rate of the reference evaluation window, or the accumulated amount of direction change increases by more than a preset increase rate compared to the accumulated amount of the reference evaluation window, the safety management module (220) can set the local instability flag to true. The local instability flag may indicate a state in which the hand object stays near the boundary line of the area of interest longer than usual, changes direction more frequently, or repeats entry and retreat more frequently.
[0207] The safety management module (220) can determine a macro misalignment state between the repetition cycle of the manufacturing equipment and the input / retrieval rhythm of the worker's hand object using the phase concentration and the retrieval margin sub-quantile value calculated in the current evaluation window. In one embodiment, if the phase concentration decreases below a preset first reference value, it can be determined that the input timing of the hand object is not maintained consistently within the equipment cycle and is dispersed. Additionally, if the retrieval margin sub-quantile value decreases below a preset second reference value, it can be determined that the minimum margin time secured from the retrieval of the hand object until the next equipment descent has decreased. If at least one of the decrease in the phase concentration or the decrease in the retrieval margin sub-quantile value is confirmed, the safety management module (220) can set the macro misalignment flag to true.
[0208] The safety management module (220) can classify the worker-equipment synchronization state in stages according to the combination of the local instability flag and the macro misalignment flag. In one embodiment, if the local instability flag is true and the macro misalignment flag is false, the safety management module (220) can set a caution state. If the local instability flag is false and the macro misalignment flag is true, the safety management module (220) can set a warning state. If both the local instability flag and the macro misalignment flag are true, the safety management module (220) can set a danger state. The danger state can be used as a state that requires a preliminary alarm output even before an actual valid intrusion event is confirmed.
[0209] The safety management module (220) may set different preliminary alarm stages corresponding to the attention state, the warning state, and the danger state. In one embodiment, the attention state may correspond to a first preliminary alarm stage, the warning state may correspond to a second preliminary alarm stage, and the danger state may correspond to a third preliminary alarm stage. The first preliminary alarm stage may be implemented as a visual display-centered warning, such as a change in screen border color or a gentle flashing display. The second preliminary alarm stage may be implemented as a visual display that is more emphasized than the first preliminary alarm stage or as a warning mark on the administrator screen. The third preliminary alarm stage may be implemented in a form that combines a warning sound or a rapid flashing cycle with a visual display.
[0210] The preliminary alarm according to the above process 3 can be distinguished from the valid intrusion event according to the above process 1. The preliminary alarm does not mean that the hand object is included within the area of interest for more than a number of consecutive reference frames, but rather indicates that the input / retrieval rhythm of the worker object in multiple equipment cycles is changing differently from the reference state. Therefore, even if the preliminary alarm is output, the safety management module (220) can continue to separately perform the determination of the valid intrusion event according to the above process 1. When the valid intrusion event is confirmed, the notification output unit (300) can output a warning of the main alarm stage, and the preliminary alarm stage can be stored separately from the main alarm stage.
[0211] The safety management module (220) can set reference values for the number of hesitations, the dwell rate of the inner band area of the boundary line, the accumulated amount of direction change, the phase concentration, and the lower quantile value of the recovery margin using normal operation data from the initial installation or an operation section designated as normal operation by the manager. In one embodiment, the reference values for the number of hesitations, the dwell rate of the inner band area of the boundary line, and the accumulated amount of direction change can be set by adding an allowable increase margin to the average value or median value of the normal operation evaluation window. The first reference value for the phase concentration can be set based on the lower allowable range of the phase concentration calculated in the normal operation section, and the second reference value for the lower quantile value of the recovery margin can be set by subtracting a repair margin from the lower quantile value of the normal operation section. The reference values can be updated according to the operator's proficiency, the number of strokes per minute of the equipment, the repeatability of the operation process, and the on-site safety operation policy.
[0212] In one embodiment, the recovery margin frames measured during the last 10 cycles in a normal operating state may appear at the levels of 22 frames, 24 frames, 19 frames, 21 frames, 23 frames, 20 frames, 18 frames, 25 frames, 21 frames, and 19 frames. When the values are sorted in ascending order, they may be arranged as 18 frames, 19 frames, 19 frames, 20 frames, 21 frames, 21 frames, 22 frames, 23 frames, 24 frames, and 25 frames. In this case, the safety management module (220) may identify the lower quantile range of the normal rhythm section as being around 18 to 19 frames, and to secure more on-site safety margin, the second reference value may be set to the level of 17 frames by subtracting a repair margin of 1 frame from the lower quantile range. Accordingly, if the recovery margin lower quantile value in the current evaluation window decreases to 17 frames or less, the safety management module (220) can set the macro misalignment flag to true.
[0213] In one embodiment, there may be cases where the number of hesitations per evaluation window in the normal operation section is at an average level of 1, the dwell rate in the inner band area of the boundary line is at an average level of 10 percent, and the cumulative amount of direction change is calculated at an average level of 40 degrees. At this time, the safety management module (220) may set the increase criterion for the number of hesitations to 1.5 times the reference average, set the increase criterion for the dwell rate in the inner band area of the boundary line to increase by 5 percentage points or more compared to the reference average, and set the increase criterion for the cumulative amount of direction change to increase by 20 degrees or more compared to the reference average. Accordingly, if the number of hesitations in the current evaluation window increases to 2 times or more, the dwell rate in the inner band area of the boundary line increases to 15 percent or more, or the cumulative amount of direction change increases to 60 degrees or more, the safety management module (220) may set the local instability flag to true.
[0214] In the above normal rhythm scenario, the input start time of the hand object may be located relatively repeatedly within a certain section of the manufacturing equipment cycle, and the recovery margin lower quantile value may be maintained at a value greater than the second reference value. Additionally, the number of hesitations, the retention rate of the inner band area of the boundary line, and the accumulated amount of direction change may each remain within the allowable range of the reference evaluation window. In this case, the safety management module (220) may set both the local instability flag and the macro misalignment flag to false and not output a preliminary alarm.
[0215] If the time of retrieval of the hand object is gradually delayed due to the accumulation of fatigue of the above-mentioned worker or a decrease in work concentration, the above-mentioned retrieval margin lower quantile value may decrease to below the above-mentioned second reference value. For example, if the retrieval margin lower quantile value is calculated to be at the 15-frame level in the current evaluation window, since the value is below the previously set second reference value of 17 frames, the above-mentioned safety management module (220) can set the macro misalignment flag to true. At this time, if the above-mentioned number of hesitations, the above-mentioned inner band area dwell rate, and the above-mentioned accumulated amount of direction change remain within the normal range, the above-mentioned safety management module (220) can set a warning state and control to output a second preliminary alarm step.
[0216] If the retrieval margin of the hand object decreases while the number of hesitations or the accumulated amount of direction changes near the boundary line of the area of interest increases, the safety management module (220) may set both the local instability flag and the macro misalignment flag to true. For example, if the lower quantile value of the retrieval margin in the current evaluation window decreases to 15 frames, the number of hesitations increases to more than 1.5 times the reference average, and the dwell rate of the inner band area of the boundary line also exceeds the reference value, the safety management module (220) may classify the state as a dangerous state. In this case, the notification output unit (300) may provide a visual and auditory display in parallel in response to the third preliminary alarm stage, or provide a highlight display on the administrator screen through the monitoring module (510).
[0217] Since the above process 3 evaluates the worker-equipment synchronization status based on normal continuous production, separate exception handling may be applied in operation modes different from normal production, such as mold replacement, test run, setting work, maintenance work, or worker training periods. The above safety management module (220) can identify the above irregular operation mode based on manager setting information, operation mode input information, or equipment status linkage information. When the above irregular operation mode is identified, the above safety management module (220) can temporarily suspend the preliminary alarm judgment of the above process 3 or control the above standard evaluation window not to be updated. Accordingly, the phenomenon in which work patterns different from the normal production rhythm are repeatedly classified as preliminary alarms can be reduced.
[0218] The safety management module (220) may exclude some trajectory data obtained from the evaluation window from statistical calculations if the coordinate tracking of a hand object in a continuous frame becomes unstable more than a certain number of times due to lens contamination, dust, reflected light, sudden changes in illumination, or temporary tracking failure. Additionally, if the tracking failure rate exceeds a preset range, the safety management module (220) may treat the evaluation window as an invalid evaluation window and maintain the state of the previous valid evaluation window or return it to the default state. In this case, the safety management module (220) may provide a notification to the client unit (500) requesting an image quality check, lens cleaning, or a reset of the area of interest.
[0219] The safety management module (220) can transmit the local instability flag, macro misalignment flag, preliminary alarm stage, evaluation window identification information, and related derived variable values calculated according to process 3 to the server unit (400). The data storage module (420) of the server unit (400) can store the time of occurrence of the preliminary alarm, equipment identification information, work zone identification information, and judgment results per evaluation window. The monitoring module (510) of the client unit (500) can display the preliminary alarm history by time period or display it for comparison with the history of valid intrusion events. Accordingly, the manager can check the history of changes in work rhythm in the stage prior to the actual intrusion event and use it as a reference for worker placement, adjustment of break times, or inspection of equipment operating conditions.
[0220] In order to control the operation of the above notification output unit (300), the edge computation unit (200) and the server unit (400) can variably adjust the alarm intensity by reflecting the field situation and the response history of the worker or manager together. Process 4 may be a procedure that, when an intrusion event occurs, does not repeatedly apply the same acoustic output or the same optical output in all situations, but rather adjusts the levels of auditory output and visual output by reflecting the entry form of the intruding object, field visibility, alarm confirmation delay, and the accumulated state of unreleased alarms.
[0221] Process 4 above may be applied when a valid intrusion event is confirmed according to Process 1 above or when a preliminary alarm stage is set according to Process 3 above. That is, the edge computation unit (200) can calculate a basic alarm intensity based on the degree of morphological entry of the intruding object, and the server unit (400) can subsequently correct the basic alarm intensity based on recent alarm history and user response history. The notification output unit (300) can determine the final alarm output parameter according to the basic alarm intensity and the subsequent correction result.
[0222] The judgment criteria used in the above process 4 may include a primary judgment criterion calculated directly from the video frame at the time of the intrusion event occurrence, and a secondary correction criterion calculated from the alarm occurrence or past alarm history. The primary judgment criterion may include a boundary line occupancy length ratio and a short-period vertical drop amount. The secondary correction criterion may include an average acknowledgment delay value, a visibility reduction rate, and an unreleased alarm rate. The primary judgment criterion may be used to evaluate the degree of the intrusion itself, and the secondary correction criterion may be used to reflect the state in which the alarm is actually recognized and processed at the site.
[0223] The edge calculation unit (200) can set the basic alarm intensity to one of the first, second, or third stages using the boundary line occupancy length ratio and the short-period vertical drop amount. Subsequently, the server unit (400) can correct the basic alarm intensity using the average acknowledgment delay value, the visibility reduction rate, and the unreleased alarm rate. At this time, the average acknowledgment delay value can be used primarily for correcting auditory output parameters, the visibility reduction rate can be used primarily for correcting visual output parameters, and the unreleased alarm rate can be used for resetting the higher stage of both auditory and visual outputs.
[0224] The above boundary line occupancy length ratio can be calculated as the ratio of the pixel length of the section where the mask, contour, or bounding box of the infringing object touches or overlaps with the boundary line of the area of interest, relative to the total boundary line length of the area of interest. In one embodiment, the edge calculation unit (200) can calculate the boundary line occupancy length ratio by dividing the boundary line of the area of interest into a plurality of boundary line pixels or boundary line segments, accumulating the length of the boundary line pixels or segments that overlap with the infringing object, and dividing the result by the total boundary line length of the area of interest. If the boundary line occupancy length ratio increases, it can be interpreted as a state where a wider portion of the object, rather than a local infringement such as a fingertip, extends across the boundary line of the danger zone.
[0225] The edge calculation unit (200) can classify a case where the boundary line occupancy length ratio is less than a first occupancy standard as a low occupancy section, a case where the boundary line occupancy length ratio is greater than or equal to the first occupancy standard but less than the second occupancy standard as an intermediate occupancy section, and a case where the boundary line occupancy length ratio is greater than or equal to the second occupancy standard as a high occupancy section. The second occupancy standard can be set to a value greater than the first occupancy standard. Accordingly, the edge calculation unit (200) can distinguish the degree to which an intruding object contacts the boundary line of the area of interest in stages.
[0226] The above short-period vertical drop amount can be calculated based on the pixel difference value of the vertical movement of the representative coordinates of the intruding object between a plurality of consecutive image frames. In one embodiment, the representative coordinates of the intruding object may be set as the center point, bottom center point, or the object boundary point that has entered the deepest part of the region of interest of the intruding object bounding box. The edge calculation unit (200) can calculate the vertical movement amount of the representative coordinates between the current frame and the previous frame or between a predetermined few previous frames, and calculate the short-period vertical drop amount by accumulating the downward direction component of the vertical movement amount or extracting it as a maximum value.
[0227] When the above short-period vertical descent amount is small, the intruding object may be interpreted as moving relatively gently or being in a state close to stopping near the boundary line. On the other hand, when the above short-period vertical descent amount is large, the intruding object may be interpreted as moving rapidly downward for a short period of time, or as entering the direction of the area of interest while part of the worker's body is tilted or leaning. The edge calculation unit (200) can divide the above short-period vertical descent amount into a low descent section, a medium descent section, and a high descent section, and can reflect the result of the division in the calculation of the basic alarm intensity.
[0228] The edge calculation unit (200) can calculate a basic alarm intensity by evaluating the boundary line occupancy length ratio and the short-period vertical descent amount together. In one embodiment, if the boundary line occupancy length ratio is in a low occupancy section and the short-period vertical descent amount is in a low descent section, a basic alarm intensity of the first stage may be set. If either the boundary line occupancy length ratio or the short-period vertical descent amount corresponds to an intermediate section, a basic alarm intensity of the second stage may be set. Additionally, if both the boundary line occupancy length ratio and the short-period vertical descent amount correspond to a high section, or if either one exceeds a pre-set upper standard, a basic alarm intensity of the third stage may be set.
[0229] The basic alarm intensity of the first stage above may be implemented as at least one of a short warning sound of low volume, a gentle flashing, or an on-screen warning display. The basic alarm intensity of the second stage above may be implemented as at least one of a higher volume than the first stage, a shorter flashing cycle, or an emphasized on-screen display. The basic alarm intensity of the third stage above may be implemented as a higher volume than the second stage, a shorter flashing cycle, increased light intensity, or a parallel output of visual and auditory displays. Each of the above stages is not limited to an absolute volume value or a specific light intensity value, but may be set as a stage value according to the device specifications and field operation standards of the notification output unit (300).
[0230] The above average value of confirmation delay can be calculated as an average value after collecting the time difference between the time of alarm occurrence and the time of user confirmation or release operation for a recently pre-set number of alarm events or alarm events within a recently pre-set time window. In one embodiment, the time of alarm occurrence may be the time when the notification output unit (300) starts outputting a warning sound or flashing a warning light. The time of user confirmation or release operation may be the time when an administrator presses the confirmation button on the dashboard screen of the client unit (500), the time when the field release button is entered, or the time when an alarm confirmation operation is entered on a mobile terminal. If the above average value of confirmation delay increases, it may be interpreted as a state where the alarm is recognized or processed late in the field.
[0231] The server unit (400) may upwardly correct an auditory output parameter corresponding to the basic alarm intensity when the average value of the confirmation delay exceeds a preset first correction standard. In one embodiment, the auditory output parameter may include a volume level, the number of warning sound repetitions, the duration of the warning sound, or the warning sound repetition cycle. For example, even if the basic alarm intensity is set to the first level, if the average value of the confirmation delay exceeds the first correction standard, the server unit (400) may transmit a control signal to the notification output unit (300) to upwardly increase only the auditory output to a level corresponding to the second level.
[0232] The above visibility reduction rate can be calculated based on the background luminance and color distribution around the location where the notification is displayed, and the contrast value between the output color of the warning light or screen display. In one embodiment, the edge calculation unit (200) or the server unit (400) may calculate the background pixel brightness value around the area where the warning light is located or the area where the screen warning is displayed within the video frame, and calculate the brightness contrast value using the difference between the background pixel brightness value and the reference brightness value of the warning light color. Additionally, the color difference between the background color and the warning light color may be reflected secondarily. If the above visibility reduction rate increases, the visual warning may be interpreted as being difficult to distinguish from the surrounding background or lighting conditions.
[0233] The server unit (400) may increase the visual output parameter corresponding to the basic alarm intensity when the visibility reduction rate exceeds a preset second correction standard. In one embodiment, the visual output parameter may include the flashing cycle of the warning light, the light intensity level, the display color, the screen popup size, the screen border flashing frequency, or the contrast level of the warning display. For example, if the visibility reduction rate is calculated to be high because the ambient illumination is high or the contrast between the background color and the warning display color is low, the server unit (400) may correct the visual output by shortening the flashing cycle of the notification output unit (300) or by further emphasizing the screen warning display of the monitoring module (510).
[0234] The above unresolved alarm ratio can be calculated as the ratio of alarms occurring at the same facility and location for which a normal verification or release procedure was not performed, based on a recently configured time window or a plurality of recently configured alarm events. The above normal verification or release procedure may include at least one of an administrator confirmation input, a field release button input, a cause inspection completion input, or an alarm status resolution confirmation input. If the above unresolved alarm ratio increases, it may be interpreted as a state in which repeated alarms at the same facility or location are not properly processed and are accumulating.
[0235] The server unit (400) may reset the final alarm intensity to a higher level regardless of the individual correction results based on the average confirmation delay value and the visibility reduction rate when the unreleased alarm rate exceeds a preset third correction standard. In one embodiment, when the unreleased alarm rate is greater than or equal to the third correction standard, the server unit (400) may reset both the auditory output and the visual output to a third level or a higher level allowed by the notification output unit (300). Accordingly, a separate priority rule may be applied to increase both the auditory output and the visual output for situations where alarms are repeatedly left unattended.
[0236] After the basic alarm intensity is calculated, the server unit (400) can independently perform auditory correction based on the average value of the confirmation delay and visual correction based on the visibility reduction rate. Subsequently, the server unit (400) can determine whether the unreleased alarm rate exceeds the third correction standard. If the unreleased alarm rate does not exceed the third correction standard, the auditory correction result and the visual correction result can be used as the final alarm output value. On the other hand, if the unreleased alarm rate exceeds the third correction standard, the final alarm output value can be reset to a higher level regardless of the individual correction results.
[0237] The notification output unit (300) can determine final output parameters using the basic alarm intensity and the subsequent correction result. The final output parameters may include auditory output parameters and visual output parameters. The auditory output parameters may include a volume level, the number of warning sound repetitions, the duration of the warning sound, or the warning sound repetition cycle, and the visual output parameters may include a flashing cycle, a light intensity level, a color change, a screen display size, or a screen highlighting level. The notification output unit (300) can control a buzzer, a speaker, a lamp, a warning light, a field display device, or a screen display of the client unit (500) according to the final output parameters.
[0238] In one embodiment, the first occupancy criterion for the boundary line occupancy length ratio may be set to a level of 20 percent, and the second occupancy criterion may be set to a level of 50 percent. Additionally, the first descent criterion for the short-period vertical descent amount may be set to a level of 15 pixels, and the second descent criterion may be set to a level of 40 pixels. In this case, an intrusion event in which the boundary line occupancy length ratio is 10 percent and the short-period vertical descent amount is 8 pixels may be classified as a basic alarm intensity of the first stage. On the other hand, if the boundary line occupancy length ratio is 35 percent or the short-period vertical descent amount is 25 pixels, it may be classified as a basic alarm intensity of the second stage. Additionally, if the boundary line occupancy length ratio is 60 percent and the short-period vertical descent amount is 45 pixels, it may be classified as a basic alarm intensity of the third stage.
[0239] In one embodiment, the time taken for a worker or manager to check or clear an alarm for the last 10 alarm events can be measured at levels of 3 seconds, 4 seconds, 3 seconds, 2 seconds, 4 seconds, 2 seconds, 4 seconds, 3 seconds, 5 seconds, and 3 seconds. The average checking delay time for the 10 events can be calculated as approximately 3.3 seconds. The server unit (400) can set the first correction standard to a level of 6 seconds by adding the allowable time required for field movement or screen checking to the average response time of the normal operation period. In this case, if the checking or clearing operation is not performed for more than 6 seconds after a specific alarm event occurs, the server unit (400) can determine that the upward correction condition for the auditory output parameter is satisfied.
[0240] In one embodiment, the visibility reduction rate may be normalized to a value between 0 and 1. As the visibility reduction rate approaches 0, the visual warning may be interpreted as being relatively well distinguished from the background, and as the visibility reduction rate approaches 1, the visual warning may be interpreted as being difficult to distinguish from the background. For example, if the server unit (400) calculates the visibility reduction rate to be 0.7 or higher, it may determine that the second correction criterion has been exceeded. In this case, the notification output unit (300) may shorten the flashing cycle of the warning light or increase the light intensity level, and the monitoring module (510) may correct the visual warning by increasing the flashing frequency or color contrast of the screen warning border.
[0241] In one embodiment, the unreleased alarm ratio may be calculated based on alarm events that occurred at the same facility and at the same location during the last 10 minutes. For example, if 5 alarm events occurred at the same location during the last 10 minutes and normal verification or release procedures were not performed for 3 of them, the unreleased alarm ratio may be calculated as 60 percent. If the third correction standard is set to a level of 50 percent, the server unit (400) determines that the unreleased alarm ratio has exceeded the third correction standard and may reset both the auditory output and the visual output to a higher level.
[0242] In the above minor intrusion scenario, a small object part, such as a fingertip, may come into contact with the boundary of the region of interest for a short period of time, and the short-period vertical drop amount may also remain in a low range. In this case, the edge calculation unit (200) can set the basic alarm intensity of the first stage. If the average value of the acknowledgment delay, the visibility reduction rate, and the unreleased alarm rate are all below the corresponding correction standard, the notification output unit (300) can provide a low auditory output or a gentle visual display corresponding to the first stage. Accordingly, the situation in which excessive output is repeated for minor intrusions can be reduced.
[0243] If the user does not perform a confirmation or release operation within a preset time after the above intrusion event occurs, the server unit (400) may determine that the average value of the confirmation delay or the confirmation delay time of the event has exceeded the first correction criterion. For example, if the basic alarm intensity is set to the first level but no confirmation or release operation is performed until 8 seconds have elapsed after the alarm occurs, the server unit (400) may correct the auditory output parameter upward to the second level. In this case, the visual output parameter may be maintained at the basic level unless a separate visibility reduction condition is satisfied.
[0244] If the illuminance of the workplace is high or the background around the warning light is similar to the color of the warning light display, making it difficult to distinguish the visual warning, the visibility reduction rate may exceed the second correction standard. In this case, the server unit (400) may correct the visual output parameter upward. For example, if the visibility reduction rate is calculated to be 0.7 or higher while the basic alarm intensity is set to the second level, the notification output unit (300) may shorten the flashing cycle of the warning light or increase the light intensity level, and the monitoring module (510) may increase the size of the warning popup or the flashing frequency of the border.
[0245] If an alarm event occurs repeatedly at the same facility and location but the confirmation or release procedure is not sufficiently performed, the unreleased alarm rate may exceed the third correction standard. In this case, the server unit (400) may reset the final alarm intensity to a higher level regardless of the individual correction results based on the average confirmation delay value or the visibility reduction rate. For example, even if the basic alarm intensity is set to the first or second level, if the unreleased alarm rate is calculated to be above the standard, both the auditory output and the visual output may be switched to the third level or a higher level allowed by the device.
[0246] The above notification output unit (300) can be controlled so as not to exceed a preset safety output upper limit or device output upper limit, even if the auditory output or the visual output is repeatedly corrected upward. In one embodiment, the auditory output parameter may be limited to a level lower than the maximum sound level set by field operation standards or device specifications, and the visual output parameter may be limited to a level lower than the allowable light level of a lamp or warning light. Accordingly, the burden on the device due to excessive output settings or inappropriate alarm output in the working environment can be reduced.
[0247] If the number of alarm events required to calculate the average value of the confirmation delay falls short of a preset minimum number, the server unit (400) may maintain the default auditory output value without performing the correction based on the average value of the confirmation delay. Additionally, if the number of alarm events required to calculate the unreleased alarm ratio is excessively low, the unreleased alarm ratio may be excluded from the correction judgment. If the image data required to calculate the visibility reduction rate is unreliable due to reflected light, white balance errors, image acquisition failure, or camera noise, the server unit (400) may withhold visual correction based on the visibility reduction rate and apply the default visual output value or a preset repair value.
[0248] The server unit (400) can store the basic alarm intensity, average acknowledgment delay value, visibility reduction rate, unreleased alarm ratio, final alarm intensity, and output parameters calculated according to the process 4 in the data storage module (420). The data storage module (420) can store the alarm event occurrence time, facility identification information, work zone identification information, user acknowledgment time, release status, and output stage change history together. The monitoring module (510) of the client unit (500) can use the stored alarm stage history to display sections where alarms are repeatedly raised in the same facility, time periods where acknowledgment delays accumulate, or work zones where time warning identification appears low.
[0249] According to the above process 4, different levels of alarm output may be applied to the same intrusion event depending on the degree of entry of the intruding object, user response history, and on-site visibility conditions. Accordingly, a relatively low level of output may be provided for minor intrusions, and auditory and visual outputs may be operated in a direction of gradually increasing when confirmation delay, reduced visual warning identification, or repeated neglect is confirmed. In addition, by applying the above output upper limit and data error exception handling together, situations in which the upward correction of the alarm output is excessively applied due to device limitations or incomplete input data can be reduced.
[0250] The operation of the vision edge-based process safety and production integrated management system according to the present invention is described below using a manufacturing site equipped with press equipment as an example. This embodiment is an example to aid in understanding the present invention, and the scope of application of the present invention is not limited to press equipment. However, the following embodiment is based on an environment where the manufacturing equipment is a press equipment that repeatedly has an upward state, a downward state, a bottom-reaching state, and a return-completed state, and where an operator performs product feeding or retrieval operations in front of the press equipment.
[0251] The image acquisition unit (100) may be installed so that the slide of the press equipment, the upper mold, the lower mold, the access path of the worker's hand, and the danger zone are included within a single shooting range. In one embodiment, the image acquisition unit (100) acquires at least 30 frames of image per second and may use an infrared irradiation function during low-light periods. The edge calculation unit (200) receives the image frames and can set a region of interest, an equipment reference point, an area between molds, and a worker object detection area, respectively.
[0252] The above system can collect normal operation data and normal impact data during the calibration phase at the initial installation. The normal operation data may include the worker object position, hand object trajectory, point of entry into the region of interest, point of retrieval, and equipment cycle data while the worker performs product feeding and retrieval operations at a normal speed and movement path. The normal impact data may include the arm connection width, number of hang frames, edge pattern fixation rate, descent symmetry error, and fine rebound displacement data while the press equipment descends normally to perform product impact.
[0253] When the image acquisition unit (100) acquires a series of image frames, the edge calculation unit (200) can perform at least one of resolution adjustment, region of interest-based cropping, and noise reduction on the image frames. Subsequently, the object detection module (210) calculates a bounding box of the worker object or hand object, and the safety management module (220) can determine the positional relationship between the bounding box and the region of interest. At the same time, the production counting module (230) can identify a state transition candidate using the slide position of the press equipment, the mold boundary, and the change in the dark area between the molds.
[0254] When the safety management module (220) generates an intrusion event or a preliminary alarm event, or when the production count module (230) generates a result of increasing the production count or holding the count, the edge computation unit (200) can transmit the result data to the server unit (400). The server unit (400) can transmit the result data to the client unit (500) through the data relay module (410) and can store the time of event occurrence, equipment identification information, related frame identification value, conditional judgment result, and user verification history through the data storage module (420). The monitoring module (510) of the client unit (500) can superimpose the location of the worker object, the boundary line of the area of interest, the intrusion event, the preliminary alarm stage, and the accumulated production count onto the real-time video.
[0255] In one embodiment, the safety management module (220) may set the number of basic reference frames for confirming an intrusion event to 5 frames. The number of basic reference frames may be an initial value for confirming a valid intrusion event based on the case where the state in which a worker object is included within the region of interest is maintained for 5 consecutive frames. However, the safety management module (220) may not apply the number of basic reference frames as is, but may subtract or add the number of reference frames according to phase proximity, boundary line intersection ratio, direction alignment, return delay, and occlusion transition frequency.
[0256] The safety management module (220) can repeatedly collect the phase proximity of a normal worker during the initial calibration period. For example, the phase proximity measured in the first 10 cycles of a normal worker may appear at levels of 0.65, 0.68, 0.62, 0.70, 0.67, 0.69, 0.66, 0.64, 0.68, and 0.71. The average value based on the 10 sample data may be calculated to be approximately 0.67, and the safety management module (220) may add an upper allowable margin to the average value to set a first threshold value for the phase proximity to approximately 0.75. Accordingly, if the phase proximity exceeds 0.75, it may be determined that the entry point of the hand object is close to the latter part of the equipment lowering section.
[0257] In one embodiment, the safety management module (220) sets the last 10 frames as an evaluation window and can calculate the number of frames in which the hand object bounding box overlaps with the boundary line of the region of interest by more than a preset minimum intersection length within the evaluation window. For example, if overlap exceeding the minimum intersection length occurs in 6 frames out of the last 10 frames, the boundary line intersection ratio can be calculated as 0.6. Additionally, if the movement vector of the hand object relatively matches the center direction of the region of interest, the direction alignment degree can be determined to exceed a corresponding threshold value. The safety management module (220) can reflect the determination results of the boundary line intersection ratio and the direction alignment degree in whether to subtract the reference frame count.
[0258] The safety management module (220) can collect the number of frames from the time a hand object enters the area of interest until it completely exits the area of interest in a recent multiple equipment cycle. For example, if the median value of the number of frames in the normal operation section is 8 frames and the median value increases to 13 frames in the current operation section, the safety management module (220) can determine that the return delay has exceeded the corresponding threshold. Meanwhile, if the overlap state of the hand object bounding box and the equipment slide bounding box in the recent evaluation window repeatedly switches between a non-overlapping state and an overlapping state, the safety management module (220) can determine that the occlusion switching frequency has exceeded the corresponding threshold.
[0259] The safety management module (220) can calculate the number of items that exceed a threshold among phase proximity, boundary line intersection ratio, direction alignment, and return delay. For example, if the phase proximity, boundary line intersection ratio, and direction alignment each exceed a corresponding threshold, and the return delay is below the threshold, the safety management module (220) can calculate the number of high-risk condition fulfillments as 3. In this case, the safety management module (220) can calculate the current reference frame number as 2 frames by subtracting 3 frames from the basic reference frame number of 5 frames.
[0260] After the above subtraction operation, the safety management module (220) can check the occlusion transition frequency. For example, if the overlap state transition of the hand object and the equipment slide in the recent evaluation window occurs at a preset fifth threshold but below the sixth threshold, the safety management module (220) can add one frame to the reference frame count. Accordingly, the two frames calculated from the previous subtraction result can be corrected to three frames. On the other hand, if the occlusion transition frequency is repeated at or above the sixth threshold, two frames can be added to correct it to four frames.
[0261] In the fixed 5-frame method, even if the worker hand object rapidly approaches the latter part of the equipment descent, a valid intrusion event may not be confirmed until the state of inclusion within the region of interest is maintained for 5 frames or more. On the other hand, in this embodiment, when the phase proximity, boundary line intersection ratio, and direction alignment all exceed threshold values, the reference frame count can be adjusted to the level of 2 or 3 frames, so the system can be operated in a way that advances the alarm confirmation time for entry states close to the danger zone. In addition, in video instability situations where the occlusion switching between the hand object and the equipment object is repeated, the reference frame count can be extended again according to the frequency of occlusion switching, so the system can be operated in a way that reduces the occurrence of false alarms based on one-time overlap or detection shake.
[0262] The above production coefficient module (230) can collect data for a plurality of equipment cycles in which normal impacts are repeated during the initial calibration period. The collected data may include arm connection width, number of airborne frames, edge pattern fixation rate, downward symmetry error, and fine rebound displacement. The above production coefficient module (230) can set reference values used for the first to third conditions based on the upper limit, lower limit, median, or percentile value of the normal impact data.
[0263] In one embodiment, the fine rebound displacement measured immediately after reaching the bottom during 10 normal strikes may appear at levels of 8 pixels, 11 pixels, 7 pixels, 12 pixels, 10 pixels, 9 pixels, 13 pixels, 8 pixels, 10 pixels, and 11 pixels. The average value of the sample is approximately 9.9 pixels, and the maximum value observed in the normal strike section can be summarized as 13 pixels. The production coefficient module (230) can set the allowable upper limit of the fine rebound displacement to 15 pixels by adding an allowable margin to the maximum value of the normal strike section.
[0264] In one embodiment, when the arm connection width is located in the range of 4 to 6 pixels upon reaching the bottom in the normal striking section, the production count module (230) may set the first value for the arm connection width to the level of 7 pixels by reflecting the allowable margin. Additionally, when the edge pattern fixation rate of the mold boundary in the normal striking section is located in the range of 88 percent to 95 percent, the production count module (230) may set the allowable lower limit of the edge pattern fixation rate to the level of 85 percent. The allowable range of the descent symmetry error may be set to a value of 10 pixels or around therefrom by reflecting the allowable margin when the difference in the amount of vertical descent between the left reference point and the right reference point in the normal striking section is located in the range of 5 to 9 pixels.
[0265] When the above press equipment descends normally to perform product impact, the production count module (230) can determine that the arm connection width between the upper and lower molds decreases to a value less than or equal to a first value, and that the reduced state is maintained for more than a preset number of frames. Subsequently, the production count module (230) can determine that in the frame section where the first condition is satisfied, the edge pattern fixation rate is greater than or equal to the allowable lower limit, and the descent symmetry error is less than or equal to the allowable range. Afterward, if the production count module (230) determines that the fine rebound displacement immediately after reaching the bottom is less than or equal to the allowable upper limit, the state transition candidate can be approved as a valid production cycle and the production count can be increased by 1.
[0266] When the camera shakes due to external impact or floor vibration, the equipment reference point on the video frame may be observed to momentarily approach the bottom arrival position. In a simple coordinate-based counting method, the production count may increase solely due to such a change in position. However, in this embodiment, the production count module (230) can check whether the edge pattern fixation rate drops below the allowable lower limit and whether the fine rebound displacement exceeds the allowable upper limit. For example, if the amount of reverse pixel movement is observed to be 20 pixels immediately after reaching the bottom due to camera shaking, the fine rebound displacement exceeds the allowable upper limit of 15 pixels; therefore, the production count module (230) can classify the state transition candidate as an invalid state transition and withhold the increase in the production count.
[0267] If the workshop lighting flickers temporarily or reflected light occurs around the mold, the contrast within the area of interest changes momentarily, and the dark area connection width may be calculated to be smaller than the actual value. In a method based on simple contrast changes, the momentary change in the dark area may be mistaken for a hit state. However, in this embodiment, the production count module (230) can also check whether the state in which the dark area connection width is reduced to or less than the first value is maintained for more than a preset number of frames. Therefore, if the state of reduced dark area connection width appears only in one frame or a short frame interval and is immediately resolved, it is determined that the first condition is not satisfied, and the increase in the production count may be withheld.
[0268] According to the present embodiment, the production count module (230) does not increase the production count merely by the fact that it has transitioned from a downward state to a bottom-reaching state, but can increase the production count only after confirming all of the first condition based on the arm joint connection width and the number of flight frames, the second condition based on the edge pattern fixation rate and downward symmetry error, and the third condition based on the fine rebound displacement. Accordingly, it can be operated in a way that reduces invalid counts that may occur due to camera shake or lighting changes in a simple coordinate-based method.
[0269] The safety management module (220) can measure the recovery margin frames during the last 10 cycles in a normal operating state. In one embodiment, the recovery margin frames may appear at the levels of 22 frames, 24 frames, 19 frames, 21 frames, 23 frames, 20 frames, 18 frames, 25 frames, 21 frames, and 19 frames. When the values are sorted in ascending order, they may be arranged as 18 frames, 19 frames, 19 frames, 20 frames, 21 frames, 21 frames, 22 frames, 23 frames, 24 frames, and 25 frames. The safety management module (220) can identify the lower quantile range of the normal rhythm section as around 18 to 19 frames, and to secure more safety margin at the site, it can subtract a maintenance margin of 1 frame to set the reference value for the lower quantile value of the recovery margin to the level of 17 frames.
[0270] In one embodiment, in a normal operation section, the number of hesitations per evaluation window may be calculated to be at an average level of 1, the dwell rate of the inner band area of the boundary line at an average level of 10 percent, and the cumulative amount of direction change at an average level of 40 degrees. In this case, the safety management module (220) may set the criterion for increasing the number of hesitations to 1.5 times the reference average, set the criterion for increasing the dwell rate of the inner band area of the boundary line to increase by 5 percentage points or more compared to the reference average, and set the criterion for increasing the cumulative amount of direction change to increase by 20 degrees or more compared to the reference average. Accordingly, if the number of hesitations in the current evaluation window increases to 2 or more, the dwell rate of the inner band area of the boundary line increases to 15 percent or more, or the cumulative amount of direction change increases to 60 degrees or more, the safety management module (220) may set the local instability flag to true.
[0271] In the current evaluation window, if the recovery margin lower quantile value decreases to the level of 15 frames, the number of hesitations increases to 2 or more, and the dwell rate in the inner band area of the boundary line increases to 15 percent or more, the safety management module (220) can set both the local instability flag and the macro misalignment flag to true. In this case, the safety management module (220) can classify the state as a dangerous state and generate a control signal corresponding to the third preliminary alarm stage. The notification output unit (300) can provide both visual and auditory displays, and the monitoring module (510) can highlight the preliminary alarm event on the administrator screen.
[0272] The edge calculation unit (200) can calculate the boundary line occupancy length ratio and the short-period vertical drop amount when an intrusion event occurs. In one embodiment, the first occupancy standard for the boundary line occupancy length ratio can be set to a level of 20 percent, and the second occupancy standard can be set to a level of 50 percent. Additionally, the first drop standard for the short-period vertical drop amount can be set to a level of 15 pixels, and the second drop standard can be set to a level of 40 pixels. An intrusion event in which the boundary line occupancy length ratio is 10 percent and the short-period vertical drop is 8 pixels can be classified as a basic alarm intensity of the first stage, an intrusion event in which the boundary line occupancy length ratio is 35 percent or the short-period vertical drop is 25 pixels can be classified as a basic alarm intensity of the second stage, and an intrusion event in which the boundary line occupancy length ratio is 60 percent and the short-period vertical drop is 45 pixels can be classified as a basic alarm intensity of the third stage.
[0273] In one embodiment, for the last 10 alarm events, the time taken for a worker or manager to check or clear the alarm can be measured at levels of 3 seconds, 4 seconds, 3 seconds, 2 seconds, 4 seconds, 2 seconds, 4 seconds, 3 seconds, 5 seconds, and 3 seconds. The average checking delay time for the 10 events can be calculated as approximately 3.3 seconds. The server unit (400) can set a first correction standard for checking delay to a level of 6 seconds by adding the allowable time required for field movement or screen checking to the average response time during normal operation. Even if the basic alarm intensity of a specific intrusion event is set to the first level, if no checking or clearing operation is performed until 8 seconds have elapsed after the alarm occurs, the server unit (400) can correct the auditory output parameter upward to the second level.
[0274] In one embodiment, the visibility reduction rate may be normalized to a value between 0 and 1. If the visibility reduction rate is calculated to be 0.7 or higher, the server unit (400) may determine that the condition for upward correction of the visual output parameter is satisfied. Accordingly, the notification output unit (300) may shorten the flashing cycle of the warning light or increase the light intensity level, and the monitoring module (510) may increase the flashing frequency or color contrast of the screen warning border. Additionally, if five alarm events occurred at the same facility and the same location within the last 10 minutes, and normal verification or release procedures were not performed for three of them, the unreleased alarm rate may be calculated as 60 percent. If the third correction standard is set to a level of 50 percent, the server unit (400) may reset both the auditory output and the visual output to a higher level.
[0275] The above system may be operated by combining processes 1 through 4 selectively or sequentially. For example, the safety management module (220) may generate a preliminary alarm according to process 3, and if actual entry into the area of interest is confirmed, confirm a valid intrusion event according to process 1. Additionally, the production count module (230) may verify whether the production count increases according to process 2 in the same time interval, and the notification output unit (300) may adjust the alarm intensity according to process 4. The server unit (400) may store the preliminary alarm, intrusion event, production count, count hold history, confirmation delay history, and alarm output stage history generated in each of the above processes together.
[0276] In a comparative operation method using a fixed reference frame count and a simple coordinate-based production count, alarm confirmation may be delayed in situations involving rapid hand entry, or the production count may increase due to camera shake or changes in lighting. On the other hand, the system of the present embodiment adjusts the reference frame count according to risk judgment criteria and image instability judgment criteria, and regarding the production count, sequentially checks the dark area connection width, number of hover frames, edge pattern fixation rate, descent symmetry error, and minute rebound displacement. Accordingly, the system can be operated in a way that distinguishes and processes the reaction timing of safety events and the approval status of the production count based on field data.
[0277] The data storage module (420) can store in chronological order the history of intrusion events, preliminary alarms, production count increases, count suspensions, alarm confirmation delays, and alarm intensity changes that occurred in the above embodiment. The monitoring module (510) can use the stored data to display the frequency of access to the danger zone during a specific time period, changes in the production count, causes of count suspensions, and the rate of unreleased alarms. Accordingly, through the system, the manager can not only perform simple real-time monitoring but also retrospectively check the relationship between safety events and production performance by facility.
[0278] Although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols
[0279] Video acquisition unit (100) Edge calculation unit (200) Object detection module (210) Safety management module (220) Production coefficient module (230) Notification output unit (300) Server unit (400) Data relay module (410) Data storage module (420) Client unit (500) Monitoring module (510)
Claims
Claim 1 The system comprises: an image acquisition unit, which is a hardware component that captures a work area of a manufacturing facility and acquires image frames; an edge computation unit, which is a hardware component that receives the image frames from the image acquisition unit and analyzes the image frames on-site to determine whether a worker object has entered a danger zone and calculates the production count of the manufacturing facility; an alert output unit, which is a hardware component that outputs an on-site alarm based on the intrusion event determination result of the edge computation unit; a server unit, which is a hardware component that receives image-related data, intrusion event data, and production count data from the edge computation unit, and relays and stores them; and a client unit, which is a hardware component that connects to the server unit and visually displays the on-site status, intrusion events, and production count of the manufacturing facility. The edge computation unit includes: an object detection module that detects the worker object within the image frames and calculates a bounding box corresponding to the worker object; and a safety management module that determines whether the worker object has entered a danger zone by comparing the bounding box with a preset region of interest, and generates an intrusion event if the state of intrusion into the danger zone is maintained for more than a number of consecutive reference frames. A vision edge AI-based process safety and production integrated management system characterized by including: a production count module that determines the operating state of the manufacturing equipment from the image frame and increases the production count based on the result of validating the state transition candidate of the manufacturing equipment. Claim 2 A vision edge AI-based process safety and production integrated management system according to claim 1, wherein the image acquisition unit includes a global shutter type image sensor to reduce shutter distortion caused by repetitive movement of the manufacturing equipment and further includes an infrared irradiation function; wherein the image acquisition unit is installed such that the main body of the manufacturing equipment, a danger zone, and a worker movement path are included within a single shooting range, and the shooting range is used as a reference image area for setting the area between the region of interest, the equipment reference point, and the mold; wherein the object detection module adjusts the image frame to a model input format, detects the worker object corresponding to the person class using a learned object detection model, and calculates the bounding box; wherein the safety management module sets one of the bottom center point of the bounding box, the reference point closest to the region of interest, or a sub-area corresponding to the hand object as an intrusion judgment reference point, determines frame by frame whether the intrusion judgment reference point is included within the region of interest, and generates the intrusion event only when the continuous inclusion state is maintained for more than the number of reference frames by accumulating the frame-by-frame judgment results. Claim 3 In claim 2, the safety management module sets a sub-region corresponding to the hand area among the bounding boxes as a tracking target, and, instead of maintaining the reference frame count as a fixed single value, sets a basic reference frame count as an initial value and then variably calculates the reference frame count based on a high-risk judgment criterion and an image instability judgment criterion, wherein the high-risk judgment criterion comprises: a phase proximity indicating where within the descent section the point at which the tracking target first crosses the boundary line of the region of interest exists, based on the total number of frames in the descent section from the point at which the slide of the manufacturing equipment begins to descend in one equipment operation cycle until the point at which it reaches the bottom; a boundary line intersection ratio indicating the ratio of frames in which the tracking target overlaps with the boundary line of the region of interest by more than a preset minimum intersection length or a preset minimum intersection pixel count during a preset multiple frame section; and a direction alignment ratio indicating the degree of alignment between the center point movement trajectory vector of the tracking target and a target vector directed from the center point of the tracking target toward the center point of the region of interest. and for a plurality of pre-set facility cycles, a return delay calculated as a representative value of the number of frames required from the point when the tracking target enters the area of interest until the point when it completely exits the area of interest;The image instability judgment criterion includes an occlusion transition frequency representing the number of times the state of the intersection area between the tracking target and the bounding box corresponding to the slide of the manufacturing equipment switches between a non-overlapping state and an overlapping state between consecutive image frames; the safety management module calculates the number of items exceeding a threshold by comparing the phase proximity, the boundary line intersection ratio, the direction alignment, and the return delay with corresponding first, second, third, and fourth thresholds, respectively; if there is one item exceeding the threshold, subtract one frame from the basic reference frame count; if there are two items exceeding the threshold, subtract two frames from the basic reference frame count; if there are three or more items exceeding the threshold, subtract up to three frames from the basic reference frame count; and if the occlusion transition frequency exceeds the fifth threshold, add one or two frames to the subtracted reference frame count, and correct the reference frame count after the subtraction and addition are performed to a preset lower and upper limit to obtain a final reference A vision edge AI-based process safety and production integrated management system characterized by calculating the number of frames. Claim 4 delete Claim 5 In claim 1, the production count module analyzes the position change, contour change, or reference area occupancy state of the manufacturing equipment in the image frame to classify the operating state of the manufacturing equipment into one of an upward state, a downward state, and a bottom-reaching state, and identifies a point in time when the state of the manufacturing equipment transitions from the downward state to the bottom-reaching state as a state transition candidate, but does not increase the production count merely on the fact that the state transition candidate has been identified, and for the state transition candidate, a first condition including whether the arm connection width between the upper mold and the lower mold has decreased to a preset first value or less during the mold closing process, and whether the state in which the arm connection width has decreased to a preset first value or less has been maintained for a preset number of floating frames or more; and a second condition including whether the edge pattern fixation rate of the mold boundary in the frame section where the first condition is satisfied is greater than or equal to a preset allowable lower limit, and whether the descent symmetry error calculated as the absolute value of the difference in vertical pixel descent amount between the left reference point and the right reference point is less than or equal to a preset allowable range. and a third condition including whether, after the first and second conditions are satisfied, the fine rebound displacement calculated as the reverse pixel movement amount of the equipment reference point in a preset multiple frame section immediately after reaching the bottom is less than or equal to a preset allowable upper limit;A vision edge AI-based process safety and production integrated management system characterized by sequentially determining, determining that one valid production cycle has been completed and increasing the production count by 1 only when the first condition, the second condition, and the third condition are all satisfied, transitioning to a count hold state without performing subsequent operations on the second and third conditions if the first condition is not satisfied, maintaining the count hold state without performing subsequent operations on the third condition if the first condition is satisfied but the second condition is not satisfied, and classifying the state transition candidate as an invalid state transition and not increasing the production count if the first and second conditions are satisfied but the micro rebound displacement in the third condition exceeds the allowable upper limit. Claim 6 In claim 5, the production count module limits the next increase in production count until it is confirmed that the manufacturing equipment transitions from the bottom reaching state to the upward state after the production count has been increased by 1, and collects data of the arm connection width, the number of hold frames, the edge pattern fixation rate, the descent symmetry error, and the fine rebound displacement for a plurality of equipment cycles in which normal striking is repeated during the initial calibration section, wherein the first value for the arm connection width is set by adding an allowable margin to the upper limit value of the arm connection width at bottom reaching observed in the normal striking section, the number of hold frames is set based on the minimum number of frames or lower quantile value in which the arm connection width is maintained at a state of being reduced to or less than the first value in the normal striking section, the allowable lower limit of the edge pattern fixation rate is set based on the lower quantile value of the edge pattern fixation rate observed in the normal striking section, and the allowable range of the descent symmetry error is set by adding an allowable margin to the upper limit value of the difference in left-right descent amount observed in the normal striking section In addition, the allowable upper limit of the micro rebound displacement is set by adding an allowable margin to the maximum value of the micro rebound displacement observed in the normal impact section, and if the edge pattern fixation rate drops below the allowable lower limit or the micro rebound displacement exceeds the allowable upper limit due to external impact or camera shake, the state transition candidate is classified as an invalid state transition, and even if the dark area connection width temporarily decreases to below the first value due to lighting changes or reflected light, if the state in which the dark area connection width has decreased to below the first value is not maintained for more than the number of suspended frames, it is determined that the first condition is not satisfied, and due to frame omission, lens contamination, sudden change in illumination, or temporary image acquisition failure, the dark area connection width, the edge pattern fixation rate, the descent symmetry error,A vision edge AI-based process safety and production integrated management system characterized by, if it is difficult to normally calculate some of the above-mentioned micro-rebound displacements, treating the condition as not being satisfied or suspending the evaluation of state transition candidates, and if it is impossible to calculate key reference values in more than a preset number of consecutive frames, temporarily suspending the increase of the production count and providing a notification to the client unit requesting a camera status check or a region of interest reset.
Citation Information
Patent Citations
System for predicting accident based on AI
KR1020230075594A
Apparatus for AI-Based Compliance History Management and Report Generation According to Good Manufacturing Practice(GMP) Regulations
KR102946933B1
Methods and arrangements for identifying objects
US20210192162A1
