Petroleum and petrochemical scene personnel safety violation identification method based on video analysis

Abstract

The invention provides a petroleum and petrochemical scene personnel safety violation identification method based on video analysis, and relates to the technical field of petroleum and petrochemical safety monitoring, and the method comprises the steps: detecting a personnel target through obtaining video data, constructing a resident migration chain to identify a repeated round-trip region, and judging the overlapping condition with a dangerous equipment warning region. And analyzing a personnel action sequence to identify out-of-standard behaviors, verifying the compliance of the operation tool, and calculating a risk value to generate an alarm. According to the method, the safety violation behaviors in the petrochemical area can be automatically identified, the safety supervision efficiency is improved, and the safety accident risk is reduced.

Description

Technical Field

[0001] The present invention relates to petroleum and petrochemical safety monitoring technology, and in particular to a method for identifying safety violations of personnel in petroleum and petrochemical scenes based on video analysis. Background Art

[0002] Personnel safety management is crucial for ensuring production safety at petroleum and petrochemical production sites. Traditional personnel safety management relies primarily on manual inspections and fixed monitoring, which can lead to blind spots and lags in supervision, making it difficult to detect and address violations in a timely manner.

[0003] With the development of video analysis technology, video-based abnormal behavior recognition has been applied in various fields. However, existing video analysis methods primarily focus on the instantaneous behavioral characteristics of personnel and are unable to effectively identify irregular behaviors with temporal characteristics in petrochemical scenarios, such as repeated travel to and from hazardous areas and non-standard work movements. Furthermore, there is a lack of effective monitoring methods for the compliance of work tools.

[0004] Therefore, there is an urgent need for a method for identifying safety violations of personnel in petrochemical scenes based on video analysis. By analyzing the spatiotemporal behavioral characteristics of personnel targets and the use of work tools, it can realize intelligent identification of violations and timely warning, thereby improving the level of safety management in petrochemical scenes. Summary of the Invention

[0005] The embodiment of the present invention provides a method for identifying safety violations of personnel in petroleum and petrochemical scenes based on video analysis, which can solve the problems in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a method for identifying safety violations of personnel in petroleum and petrochemical scenes based on video analysis, comprising:

[0007] Obtain video surveillance data around dangerous equipment in the petrochemical area and perform target detection on the video surveillance data to obtain human targets;

[0008] A speed-time curve is established based on the position changes of the human target in consecutive video frames. The dwell points are obtained by detecting the trough points of the speed-time curve. A dwell migration chain is constructed based on the temporal relationship of the dwell points. The periodic path segments in the dwell migration chain are detected, and the areas corresponding to the periodic path segments are determined as repeated round-trip areas. The dwell duration of the repeated round-trip areas is calculated.

[0009] Determine whether the repeated travel area overlaps with the dangerous equipment warning area. If there is overlap, mark the repeated travel area as a suspicious area and calculate the initial risk value based on the residence time.

[0010] Extract the action sequence of human targets in the suspicious area and perform spatiotemporal feature analysis to obtain the action semantic description. According to the action semantic description, the non-standard actions are judged, and the initial risk value is updated for the first time based on the duration of the non-standard actions.

[0011] Identify the type of work tools carried by the target personnel and check the identification results with the work order. When unregistered work tools appear, the risk value is updated for the second time to obtain the final risk value. When the final risk value exceeds the warning threshold, alarm data is generated.

[0012] In an optional embodiment,

[0013] Obtain video surveillance data around dangerous equipment in the petrochemical area, and perform target detection on the video surveillance data to obtain human targets including:

[0014] Obtain video surveillance data around dangerous equipment in the petrochemical area and perform frame segmentation to obtain images to be processed;

[0015] Performing motion foreground analysis on the image to be processed and obtaining a moving target region through background modeling; performing adaptive threshold segmentation on the moving target region to obtain a motion foreground mask image; and extracting a target candidate region from the image to be processed based on the motion foreground mask image;

[0016] Human feature parameters are extracted from the target candidate area, including height ratio, limb length ratio and contour shape features; the target similarity score is calculated based on the human feature parameters, and the human target area is screened; the spatiotemporal position information of the human target area is extracted, including the image coordinate position and area range; a target tracker is established to perform time-series association on the human targets in consecutive frames, and output the human target containing the target number and position information.

[0017] Extracting the spatiotemporal position information of the human target area, including the image coordinate position and the area range; establishing a target tracker, performing temporal association on the human targets in consecutive frames, and outputting the human targets including the target number and position information.

[0018] In an optional embodiment,

[0019] The speed-time curve is established based on the position change of the human target in the continuous video frames. The dwell point is obtained by detecting the trough point of the speed-time curve. The following steps are included:

[0020] Obtain the position coordinates of the human target in continuous video frames, and calculate the displacement components of adjacent position coordinates in the horizontal and vertical directions respectively; calculate the displacement distance and displacement direction based on the obtained horizontal displacement component and vertical displacement component to obtain the displacement vector;

[0021] Calculating the speed and motion direction according to the displacement vector and the video frame rate; arranging the speed and motion direction in chronological order to construct an initial motion feature sequence;

[0022] Calculating the mean and standard deviation of the velocity magnitudes in the initial motion feature sequence, determining a filter window base size, and simultaneously calculating the local fluctuation intensity of the velocity magnitudes, dynamically adjusting the filter window base size to obtain an adaptive window size; performing a sliding average filter on the velocity magnitudes using the adaptive window size to obtain a smoothed velocity sequence; and constructing a velocity-time curve based on the smoothed velocity sequence;

[0023] Calculating a forward difference value and a backward difference value of the smoothed velocity sequence, and identifying a velocity trough position according to a change in the signs of the forward difference value and the backward difference value;

[0024] The motion direction at the velocity trough position is extracted, and the angle between adjacent motion directions is calculated to obtain the motion direction change value. Based on the speed value, speed duration and motion direction change value at the velocity trough position, the trough point that meets the static state judgment conditions is screened out as the residence point.

[0025] In an optional embodiment,

[0026] The dwell migration chain is constructed based on the temporal relationship of the dwell points. The periodic path segments in the dwell migration chain are detected. The areas corresponding to the periodic path segments are determined as repeated round-trip areas. The dwell time in the repeated round-trip areas is calculated.

[0027] Extracting the spatiotemporal information of the dwell points, calculating the Mahalanobis distance between the dwell points to obtain spatial proximity; calculating the time intervals of the dwell points to obtain temporal correlation; and constructing a dwell point affinity matrix based on the spatial proximity and temporal correlation.

[0028] Performing region growing on the resident points based on the affinity matrix to obtain a resident region; extracting a regional feature vector of the resident region, and constructing a resident migration chain according to a temporal relationship of the resident region;

[0029] Extracting resident migration segments from the resident migration chain using a variable-length sliding window, calculating the similarity of regional feature vectors and the regional position overlap between adjacent resident migration segments, calculating the repeatability score of the resident migration segments based on the regional feature vector similarity and the regional position overlap, and marking the resident migration segments with a repeatability score higher than a preset repeatability score threshold as periodic path segments;

[0030] The spatial areas covered by the periodic path segments are merged into a repeated round-trip area, the entry time and the exit time of the stay points in the repeated round-trip area are extracted, the single stay duration of the stay points is calculated, and the single stay durations are accumulated to obtain the total stay duration of the repeated round-trip area.

[0031] In an optional embodiment,

[0032] Determine whether the repeated travel area overlaps with the dangerous equipment warning area. If there is overlap, mark the repeated travel area as a suspicious area and calculate the initial risk value based on the residence time, including:

[0033] Obtaining a set of boundary vertex coordinates of the reciprocating area and the dangerous equipment warning area, performing convex hull fitting on the set of boundary vertex coordinates to obtain area boundary lines, and calculating the horizontal and vertical projection ranges of the reciprocating area and the dangerous equipment warning area based on the area boundary lines;

[0034] Calculating the projected intersection area of ​​the repeated round-trip area and the dangerous equipment warning area based on the horizontal projection range and the vertical projection range, calculating the spatial overlap based on the ratio of the projected intersection area to the area of ​​the repeated round-trip area, and marking the repeated round-trip area with a spatial overlap exceeding a preset overlap threshold as a suspicious area;

[0035] Obtaining equipment type information, hazardous material information, and process parameter information of dangerous equipment, calculating the equipment hazard weight, extracting the residence time in the suspicious area, statistically analyzing the residence time by time period to obtain a residence time distribution sequence, calculating the residence probability based on the residence time distribution sequence, and calculating the time entropy value based on the residence probability;

[0036] The risk base value is calculated according to the spatial overlap, the residence time and the equipment hazard weight; the risk base value is weightedly combined with the time entropy value to obtain the risk initial value.

[0037] In an optional embodiment,

[0038] Extract the action sequence of human targets in the suspicious area and perform spatiotemporal feature analysis to obtain the action semantic description. According to the action semantic description, determine the non-standard action. The first update of the initial risk value is based on the duration of the non-standard action. The following steps are included:

[0039] Acquire a video sequence of a person in a suspicious area and extract the coordinate positions of the human skeleton nodes. Calculate the displacement distance and direction of the skeleton nodes between adjacent frames to obtain the node displacement vector. Establish a local polar coordinate system based on the spatial positions of different operating devices in the industrial scene, and map the node displacement vector to the corresponding polar coordinate system to obtain an action sequence that takes into account the device orientation.

[0040] Calculating the angle and distance between the action sequence and the device operating surface to obtain spatiotemporal features; extracting the interaction pattern between the person and the device based on the spatiotemporal features, and calculating the action continuity based on the temporal changes in the interaction pattern;

[0041] Calculating the fluctuation pattern of the action continuity and extracting semantic features; constructing an action semantic description considering device interaction based on the semantic features; matching the action semantic description with a standard action library and calculating a semantic difference value;

[0042] The semantic description of an action whose semantic difference value exceeds a preset threshold is marked as an out-of-specification action; the duration of the out-of-specification action is extracted; the risk update coefficient of the out-of-specification action is calculated based on the relative position relationship and interaction intensity between the out-of-specification action and the device, and the initial risk value is updated for the first time based on the risk update coefficient.

[0043] In an optional embodiment,

[0044] Identify the type of work tools carried by the target personnel and check the identification results with the work order. If an unregistered work tool appears, the risk value is updated a second time to obtain the final risk value. When the final risk value exceeds the warning threshold, alarm data is generated including:

[0045] Collecting tool instantaneous power data and operating current signals, analyzing tool load changes based on the instantaneous power data, calculating tool start and stop frequencies based on the operating current signals, and combining the load changes and start and stop frequencies to determine tool usage status;

[0046] Acquire an image of the working area, extract the geometric dimensions and appearance features of the tool from the image of the working area, combine the use status of the tool with the geometric dimensions and appearance features, and identify the type of the tool;

[0047] Obtaining work order records, comparing the tool type with the work order, and when unregistered tools are detected, calculating the tool hazard level based on the tool's power change trend and usage frequency, and determining the distance constraint between tools based on the location distribution of the tools in the work area;

[0048] Obtaining the combustible gas concentration and ambient temperature of the operating area, determining whether there is a fire hazard when using the tool based on the combustible gas concentration and ambient temperature, and updating the last updated risk value a second time based on the determination result and the tool hazard level to obtain a final risk value;

[0049] A dynamic warning threshold is calculated based on the real-time change trend of personnel density, equipment operating status, and environmental parameters in the operating area. When the final risk value exceeds the dynamic warning threshold, alarm data containing tool information is generated.

[0050] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including:

[0051] processor;

[0052] a memory for storing processor-executable instructions;

[0053] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0054] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0055] In this embodiment, video analysis is used to automatically identify personnel safety violations in petroleum and petrochemical scenarios, thereby improving the efficiency of safety supervision in dangerous areas. Through residence point analysis and migration chain construction, it is possible to accurately identify suspicious activity areas of personnel around dangerous equipment, avoiding the limitations of traditional fixed area monitoring methods. Combined with action semantic analysis technology, it is possible to automatically analyze the behavior patterns of operating personnel and accurately judge non-standard actions. At the same time, through the identification of operating tools and work order verification, a multi-dimensional risk assessment mechanism is implemented, which greatly improves the accuracy of safety violation identification and reduces false alarms. A phased risk value update strategy is adopted, which comprehensively considers factors such as the length of personnel residence, the duration of non-standard actions, and the use of unauthorized tools, forming a complete risk quantification assessment system. The warning threshold can be flexibly adjusted according to the actual scenario, making the system highly adaptable and practical, and effectively reducing the incidence of safety accidents in the petrochemical industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a method for identifying personnel safety violations in petroleum and petrochemical scenes based on video analysis according to an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the relationship between the speed-time curve and the dwell point detection results; DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1FIG. 1 is a flow chart of a method for identifying personnel safety violations in petroleum and petrochemical scenes based on video analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] Obtain video surveillance data around dangerous equipment in the petrochemical area and perform target detection on the video surveillance data to obtain human targets;

[0062] A speed-time curve is established based on the position changes of the human target in consecutive video frames. The dwell points are obtained by detecting the trough points of the speed-time curve. A dwell migration chain is constructed based on the temporal relationship of the dwell points. The periodic path segments in the dwell migration chain are detected, and the areas corresponding to the periodic path segments are determined as repeated round-trip areas. The dwell duration of the repeated round-trip areas is calculated.

[0063] Determine whether the repeated travel area overlaps with the dangerous equipment warning area. If there is overlap, mark the repeated travel area as a suspicious area and calculate the initial risk value based on the residence time.

[0064] Extract the action sequence of human targets in the suspicious area and perform spatiotemporal feature analysis to obtain the action semantic description. According to the action semantic description, the non-standard actions are judged, and the initial risk value is updated for the first time based on the duration of the non-standard actions.

[0065] Identify the type of work tools carried by the target personnel and check the identification results with the work order. When unregistered work tools appear, the risk value is updated for the second time to obtain the final risk value. When the final risk value exceeds the warning threshold, alarm data is generated.

[0066] In an optional embodiment, obtaining video surveillance data around dangerous equipment in a petrochemical area and performing target detection on the video surveillance data to obtain a human target includes:

[0067] Obtain video surveillance data around dangerous equipment in the petrochemical area and perform frame segmentation to obtain images to be processed;

[0068] Performing motion foreground analysis on the image to be processed and obtaining a moving target region through background modeling; performing adaptive threshold segmentation on the moving target region to obtain a motion foreground mask image; and extracting a target candidate region from the image to be processed based on the motion foreground mask image;

[0069] Extracting human body feature parameters from the target candidate area, including height ratio, limb length ratio, and contour shape features; calculating a target similarity score based on the human body feature parameters, and screening to obtain the human body target area;

[0070] Extracting the spatiotemporal position information of the human target area, including the image coordinate position and the area range; establishing a target tracker, performing temporal association on the human targets in consecutive frames, and outputting the human targets including the target number and position information.

[0071] This invention provides a method for acquiring video surveillance data around hazardous equipment in a petrochemical zone and performing human target detection. First, a camera acquires video surveillance data around hazardous equipment in the petrochemical zone. The video data is typically acquired at a frame rate of 25 frames per second and a resolution of 1920×1080 pixels. The acquired video data undergoes system preprocessing, segmenting the continuous video data into individual image frames to form a sequence of images to be processed.

[0072] The image to be processed is subjected to motion foreground analysis, and a modified Gaussian mixture model (GMM) is used for background modeling. This model uses multiple Gaussian distributions to describe the time series changes of each pixel. For each pixel position (x, y), its probability density function is represented by the weighted sum of K Gaussian distributions. In this embodiment, the K value is set to 3 to 5 and adjusted according to the complexity of the scene. During the model training phase, the initial 150 frames of images are used to establish a background model. The weights, means, and variances of each Gaussian distribution are dynamically adjusted through an online update mechanism. After the background model is established, the pixel value of the current frame is compared with the background model. If the difference exceeds a preset threshold (usually set to 2.5 times the standard deviation), it is determined to be a foreground pixel.

[0073] Adaptive threshold segmentation is implemented for moving target areas, using the maximum inter-class variance method to automatically determine the optimal threshold. This method calculates the image grayscale histogram, iterates through all possible threshold values ​​(0-255), and selects the threshold that maximizes the inter-class variance as the segmentation threshold. Under complex lighting conditions, the system divides the image into a 16×16 grid and calculates a local threshold for each grid to improve segmentation accuracy. After segmentation, a binary motion foreground mask image is generated, where the foreground pixel value is 255 and the background pixel value is 0.

[0074] To reduce noise interference, morphological operations were applied to the motion foreground mask image. These included opening (erosion followed by dilation) of a 3×3 structuring element to remove small noise points and closing (dilation followed by erosion) of a 5×5 structuring element to fill internal holes in the target. After morphological processing, connected component analysis was used to extract candidate target regions. Regions smaller than 300 pixels were removed, and the remaining connected regions were marked and their minimum bounding rectangles were extracted as candidate target regions.

[0075] Within the extracted target candidate area, human feature parameters are calculated to identify human targets. The height ratio feature is obtained by calculating the target's height-to-width ratio. The height-to-width ratio of a normal upright human body is approximately 3:1 to 4:1. The system sets a higher similarity score for targets with a height-to-width ratio in the range of 2.5 to 4.5. The limb length ratio feature is based on human anatomical standards. By detecting the contour points of the candidate area and analyzing its vertical projection distribution, the positions of the head, torso, and legs are identified. In the standard human body proportions, the head accounts for approximately 1 / 8 of the total height, the torso accounts for 3 / 8, and the legs account for 4 / 8. If the body part ratios in the target area are close to the above standards, a higher similarity score is obtained.

[0076] To improve the recognition accuracy of human targets in candidate regions, the contour feature extraction stage uses an improved Histogram of Oriented Gradients (HOG) descriptor. This descriptor improves upon the traditional HOG method with structural and computational optimizations, enhancing its adaptability and robustness to human targets in complex backgrounds and various poses. First, the candidate image region is scale-normalized to a fixed size (e.g., 64×128 pixels) to ensure comparability of features extracted from targets of different scales. The normalized candidate region is then divided into an 8×8 grid, resulting in 64 sub-blocks, each of 8×8 pixels. Within each sub-block, the gradient magnitude and direction are calculated pixel by pixel. The Sobel operator is used to extract edge responses in the horizontal and vertical directions. The gradient magnitude and direction of each pixel are used to construct a directional histogram for that sub-block.

[0077] In order to improve the discriminative ability of the HOG descriptor, two improvements were introduced in the process of constructing the directional histogram. First, in terms of directional channel division, the traditional HOG method usually divides the gradient direction into 9 channels, each covering an angle range of 20 degrees. The improved scheme analyzes the distribution characteristics of edge gradients in human images and adjusts the channel center angle to make it more concentrated on common edge directions of the human body, such as vertical, oblique and horizontal directions, thereby improving the representation ability of the shoulders, limbs and torso contours. Secondly, based on the accumulation of gradient amplitudes, a position weight mechanism is introduced, that is, pixels closer to the sub-block center are given higher weights to enhance the influence of the target main contour on feature representation, while reducing the interference of background noise and edge areas.

[0078] Each sub-block ultimately generates a 9-dimensional histogram of oriented gradients (HOG) feature vector, resulting in a 64×9 (576-dimensional) HOG feature vector for the entire candidate region. To further enhance scale invariance and illumination robustness, the generated feature vector undergoes L2 normalization. The extracted HOG feature vector is then compared against a pre-built human template library for similarity. This library contains 500 standard human figures in various poses, covering common working conditions such as standing, walking, bending, carrying, and squatting. The HOG features for each template are extracted and stored using the same normalized method. The comparison uses cosine similarity as a matching metric, calculating the cosine of the angle between the candidate region's features and the template's features. If the similarity score between the candidate region and any template exceeds a set threshold (set at 0.75 in this study), the target region is considered a human. Furthermore, to avoid repeated detection, multiple highly similar candidate regions within the same image frame are screened using non-maximum suppression, ultimately outputting the optimal recognition result.

[0079] Extract the spatiotemporal position information of the identified human target area and record the center coordinates of the target in the image (c x , c y ) and the area range [x1, y1, x2, y2] (coordinates of the upper left and lower right corners). The spatiotemporal position information also includes an estimate of the target's position in the actual scene. This information is obtained by converting image coordinates into real-world coordinates using camera calibration parameters. In this embodiment, the camera is installed at a height of 5 meters and has a 30-degree viewing angle. A monocular ranging algorithm is used to estimate the distance from the target to the camera.

[0080] The target tracker is established to achieve temporal association of human targets in consecutive frames. The system adopts a method combining improved Kalman filter and deep appearance feature matching. The Kalman filter predicts the position and velocity state of the target in the next frame. The state vector contains [c x , c y , v x , v y ,w,h] six parameters, among which (c x , c y ) is the center coordinate, (v x , v y ) is the velocity vector, and (w, h) is the width and height. The process noise covariance matrix and the measurement noise covariance matrix are determined through experiments and are set to diagonal matrices with diagonal elements ranging from 0.01 to 0.1.

[0081] During the feature matching phase, color histogram features (RGB channels, 32 bins per channel) and texture features (LBP features, using an 8-neighborhood pattern with a radius of 1) are extracted from the target region to construct an appearance descriptor. Feature distances are calculated between candidate targets in the new frame and the tracked targets, and similarity is measured using the Bhattacharyya distance. The matching process comprehensively considers both predicted position and appearance similarity, with a matching threshold of 0.7 set; matches below this threshold are rejected.

[0082] For newly detected targets, the system assigns a unique target number and initializes the corresponding tracker. If a target is not detected for 10 consecutive frames, it is considered to have left the monitoring area and the tracker is terminated. The tracking process outputs human target data containing the target number and location information. The record format is [frame_id, target_id, x1, y1, x2, y2], where frame_id is the frame number, target_id is the target's unique identifier, and the last four parameters are the target's bounding box coordinates.

[0083] In this embodiment, accurate detection and dynamic tracking of personnel around hazardous equipment in petrochemical areas can be achieved, effectively improving the real-time perception of human activities. Through frame segmentation and motion foreground analysis, moving targets can be quickly located in complex backgrounds. Combined with adaptive thresholds and human feature parameter extraction, it can accurately distinguish between human and non-human targets, significantly improving detection accuracy. Furthermore, the extracted spatiotemporal position information, combined with a target tracker, enables continuous tracking and identification of human targets across frames, effectively supporting subsequent regional management and safety warnings, and improving the overall intelligence and response efficiency of the monitoring system.

[0084] In an optional embodiment, establishing a speed-time curve according to the position change of the human target in consecutive video frames, and obtaining the dwell point by detecting the trough point of the speed-time curve includes:

[0085] Obtain the position coordinates of the human target in continuous video frames, and calculate the displacement components of adjacent position coordinates in the horizontal and vertical directions respectively; calculate the displacement distance and displacement direction based on the obtained horizontal displacement component and vertical displacement component to obtain the displacement vector;

[0086] Calculating the speed and motion direction according to the displacement vector and the video frame rate; arranging the speed and motion direction in chronological order to construct an initial motion feature sequence;

[0087] Calculating the mean and standard deviation of the velocity magnitudes in the initial motion feature sequence, determining a filter window base size, and simultaneously calculating the local fluctuation intensity of the velocity magnitudes, dynamically adjusting the filter window base size to obtain an adaptive window size; performing a sliding average filter on the velocity magnitudes using the adaptive window size to obtain a smoothed velocity sequence; and constructing a velocity-time curve based on the smoothed velocity sequence;

[0088] Calculating a forward difference value and a backward difference value of the smoothed velocity sequence, and identifying a velocity trough position according to a change in the signs of the forward difference value and the backward difference value;

[0089] The motion direction at the velocity trough position is extracted, and the angle between adjacent motion directions is calculated to obtain the motion direction change value. Based on the speed value, speed duration and motion direction change value at the velocity trough position, the trough point that meets the static state judgment conditions is screened out as the residence point.

[0090] For example, image processing methods are first used to detect and track a person in consecutive video frames, obtaining the person's position coordinates in each frame. These coordinates represent the person's two-dimensional position in each frame, corresponding to the horizontal and vertical coordinates, respectively. Subsequently, based on the change in position coordinates between two adjacent frames, the horizontal and vertical displacement components are calculated. The horizontal displacement component reflects the magnitude of the person's left-right movement, while the vertical displacement component reflects the magnitude of the person's up-and-down movement. These two displacement components are combined to obtain the person's total displacement and direction of motion, which is determined by the ratio of the horizontal to vertical displacements. Combined with the video's frame rate (i.e., the number of video frames per unit time), the person's displacement per unit time is calculated, thereby determining the person's velocity while retaining the corresponding direction of motion. The velocity and direction obtained for each frame are arranged in chronological order to construct an initial motion feature sequence that fully describes the person's motion behavior over a continuous time period.

[0091] After obtaining the initial motion feature sequence, the velocity data in the sequence is statistically analyzed, and the mean and standard deviation of the velocity values ​​across the entire sequence are calculated to characterize the overall movement trend and fluctuation amplitude of the person during the observation period. Based on the mean velocity and standard deviation, the baseline size of the filtering window is set. The initial window width is used for subsequent smoothing of the velocity signal. Furthermore, to enhance sensitivity to changes in the person's velocity, the initial window size is dynamically adjusted based on the intensity of local velocity fluctuations—that is, the degree to which velocity values ​​within the velocity sequence change over a short period of time—to form an adaptive window. This adaptive window automatically adjusts its size based on velocity fluctuations: greater fluctuations result in a smaller window, and smaller fluctuations result in a larger window.

[0092] Using an adaptive window, a sliding average filter is applied to the velocity sequence to smooth the raw velocity data, reducing the interference from rapid velocity fluctuations caused by measurement errors and image jitter. This results in a smoothed velocity sequence. This smoothed velocity sequence truly reflects the actual motion state of the human target. Based on this, a velocity-time curve is further constructed for subsequent residence state determination.

[0093] To identify speed troughs during movement—the moments when a person's speed drops to its lowest point—we use forward and backward speed differences to determine the speed trend. The forward difference is the difference between the current speed value and the speed value at the next moment, while the backward difference is the difference between the current speed value and the speed value at the previous moment. If the forward difference changes from negative to positive, and the backward difference changes from negative to positive, it indicates that the speed value corresponding to the current frame is at a local minimum, indicating the presence of a speed trough. The entire speed-time curve is traversed, marking all moments that meet the trough characteristics.

[0094] For each point marked as a velocity trough, the corresponding motion direction is extracted and compared with the motion directions of the preceding and following frames. The angle between the motion directions is calculated to determine the motion direction change value for that point. The motion direction change value reflects the degree of change in the person's motion path before and after that point in time and is an important basis for determining the dwell state. The velocity value, duration, and motion direction change value corresponding to the trough point are combined to construct the determination criteria for the dwell state.

[0095] Specifically, when the speed value remains below a certain threshold for a period of time, and the direction of movement changes dramatically or frequently reverses during this period, it indicates that the person is basically stationary or moving at a low speed during this period, showing obvious dwelling behavior. In this case, the corresponding trough point is identified as a dwelling point.

[0096] In a specific work scenario, a person entered the monitoring area between 09:30 and 09:45, and the system continuously collected their coordinate information in the video. Between 09:36 and 09:39, the person's position changed very little, and their speed remained low. Furthermore, their direction of movement varied significantly during this period, with multiple reversals of direction, indicating that they were either moving around in a small area or waiting in place. The system identified multiple speed troughs during this period, and based on the direction change values ​​and duration, it ultimately determined that the period from 09:36 to 09:39 was a dwell state. The troughs that met the dwell determination criteria during this period were designated as dwell points.

[0097] Table 1 shows the results of dwell point detection based on the speed-time curve. It demonstrates the monitoring and tracking of a target person from 09:30 to 09:45. The data table clearly records six key indicators: time point, speed, direction, trough markers, direction change, and dwell status. The data shows that the target person initially moved at a moderate speed of 12.5 pixels / second, with a relatively stable direction (direction change of only 4°). At 09:36:00, the target's speed dropped significantly to 3.2 pixels / second, accompanied by a dramatic change in direction (68°). The system marked this point as the start of dwell. For approximately three minutes thereafter (09:36:00 to 09:39:10), the target maintained a low speed (1.8 pixels / second) and a direction change of 137°, indicating frequent changes in direction within a small area, consistent with typical dwell behavior. After 09:39:10, the target speed returned to 8.9 pixels / second, the direction became stable, and the system determined that the dwell state had ended.

[0098] Table 1 is the result of the dwell point detection based on the speed-time curve

[0099] In this embodiment, it is possible to accurately identify the behavior of people residing in the monitored area and effectively distinguish different motion patterns such as short stops and continuous movement. By constructing a speed-time curve and introducing an adaptive filtering window, it is possible to significantly suppress speed noise caused by camera jitter, posture changes, or short-term occlusions, thereby improving the accuracy of speed trough identification. Combined with the determination of movement direction changes and duration, it is possible to further eliminate the interference of occasional deceleration or turning behavior on residence identification, ensuring that the extraction results have higher stability and accuracy, providing a reliable foundation for abnormal behavior warning and personnel management in dangerous areas.

[0100] Figure 2This figure illustrates the relationship between the speed-time curve and the dwell point detection results. This figure illustrates the dwell point detection process and results based on the speed-time relationship. The solid line represents the smoothed speed curve, while the dashed line represents the original speed data. The horizontal axis represents time (09:30 to 09:50), and the vertical axis represents speed (0 to 30 pixels / second). The curve clearly shows that the target's speed decreases significantly between 09:36 and 09:39 (the light gray area), remaining below 5 pixels / second. Three key speed troughs are marked with red dots in the figure: at 09:36:00, 09:37:20, and 09:38:45, with speed values ​​approaching 3.2, 2.1, and 1.8 pixels / second, respectively. Notably, the smoothed speed curve effectively eliminates noise fluctuations in the original data, resulting in more accurate valley point identification. Before and after the stay area, the speed curve shows an obvious decline-trough-rise trend, which is highly consistent with the behavior pattern of personnel entering the stay state, maintaining the stay, and leaving the stay state.

[0101] In an optional embodiment, constructing a resident migration chain based on the temporal relationship of the resident points, detecting periodic path segments in the resident migration chain, determining areas corresponding to the periodic path segments as repeated round-trip areas, and calculating the resident duration of the repeated round-trip areas includes:

[0102] Extracting the spatiotemporal information of the dwell points, calculating the Mahalanobis distance between the dwell points to obtain spatial proximity; calculating the time intervals of the dwell points to obtain temporal correlation; and constructing a dwell point affinity matrix based on the spatial proximity and temporal correlation.

[0103] Performing region growing on the resident points based on the affinity matrix to obtain a resident region; extracting a regional feature vector of the resident region, and constructing a resident migration chain according to a temporal relationship of the resident region;

[0104] Extracting resident migration segments from the resident migration chain using a variable-length sliding window, calculating the similarity of regional feature vectors and the regional position overlap between adjacent resident migration segments, calculating the repeatability score of the resident migration segments based on the regional feature vector similarity and the regional position overlap, and marking the resident migration segments with a repeatability score higher than a preset repeatability score threshold as periodic path segments;

[0105] The spatial areas covered by the periodic path segments are merged into a repeated round-trip area, the entry time and the exit time of the stay points in the repeated round-trip area are extracted, the single stay duration of the stay points is calculated, and the single stay durations are accumulated to obtain the total stay duration of the repeated round-trip area.

[0106] The present invention provides a method for detecting repeated round-trip areas and calculating dwell duration based on the temporal relationship of dwell points. First, a dwell migration chain with a temporal relationship is constructed based on the identified dwell points. This migration chain is then periodically analyzed to extract repeated paths and identify relevant areas. The cumulative dwell duration of individuals within these areas is then calculated. Specifically, the dwell point data obtained from the video analysis module should contain complete spatiotemporal information. Each dwell point should include the spatial location coordinates of the individual in the image, as well as the start and end timestamps of the dwell behavior. When processing this data, the spatial proximity and temporal correlation strength between dwell points must be calculated. Spatial proximity can be measured by evaluating the distribution distance between two dwell points. This distance measurement should consider both the relative difference in location coordinates and the synergy of spatial distribution. For example, this can be achieved by normalizing the horizontal and vertical deviations and then performing a weighted fusion to form a stable spatial similarity metric. Temporal correlation strength can be measured by calculating the time difference between two dwell points, reflecting the order and temporal density of dwell behaviors.

[0107] Based on the aforementioned spatial proximity and temporal correlation, an affinity matrix for dwelling points is constructed through bidirectional fusion. Each element of this matrix represents the similarity relationship between two dwelling points, with higher values ​​indicating closer spatial proximity and greater temporal correlation between the two dwelling behaviors. By setting a reasonable affinity threshold and expanding the neighboring points starting from a particular dwelling point, a region growing algorithm can be used to identify preliminary dwelling regions. This region growing process is an iterative one, first adding the current dwelling point to the initial region set, then gradually adding points with high affinity to any dwelling point in the current set until no new points meet the expansion criteria.

[0108] After completing the division of the dwelling areas, descriptive features can be extracted for each area, including the density distribution of dwelling points within the area, average dwell time, area boundary range, and temporal ordering characteristics of dwelling points. These descriptive features are encapsulated as a regional feature vector, which serves as the basis for subsequent path similarity comparison. A dwelling migration chain is constructed based on the order of dwelling points along the timeline. This migration chain is essentially a directed chain structure that reflects the sequence of a target's movement between multiple dwelling areas. Each chain node corresponds to a dwelling area, and each connecting edge represents the movement of a person from one area to another.

[0109] After the resident migration chain is constructed, a variable-length sliding window mechanism is applied to the chain to extract resident migration segments in order to identify possible periodic path segments. The sliding window mechanism involves sequentially scanning the migration chain with varying window lengths (e.g., encompassing 3 to 5 consecutive regional nodes), recording each consecutive regional path segment as a segment. For any two migration segments, the similarity between their regional feature vectors is calculated, while the degree of overlap between the spatial regions they cover is also assessed. The similarity of regional feature vectors can be determined by comprehensively analyzing the differences in various feature dimensions. For example, if the resident regions within two segments have similar resident density, average dwell time, and boundary range, the segments are considered highly similar. Spatial overlap can be assessed based on the intersection of the bounding boxes of the regions. Specifically, if the resident regions within two segments highly overlap spatially, it can be considered that the path behaviors exhibit overlapping trends.

[0110] The above similarity and overlap are combined to form a repeatability score indicator to quantify the periodic repetitive characteristics between the two segments. By setting a repeatability score threshold, for example, setting it to a higher consistency standard, the migration segment is marked as a periodic path segment only when the repeatability score exceeds the threshold. This step effectively avoids misjudgments caused by occasional path repetitions or pseudo-periodic behaviors. After marking all periodic path segments, the spatial areas they cover can be combined into a special type of area, defined as a repeated round-trip area. This area often has frequent entry and exit or regular activities of personnel, which may be related to specific work processes or abnormal patrols and require key monitoring.

[0111] Furthermore, to calculate the duration of stays in this repeatedly traversed area, the entry and exit times of each dwell point are extracted, and the duration of each single dwell behavior is calculated. The single dwell times of all dwell points within the area are cumulatively added up to obtain the total dwell time of the person in the repeatedly traversed area. This total dwell time can be used to assess the activity level of the person in the area, the frequency of area use, or the potential risk level. For example, if a certain person frequently stays in a dangerous area and the cumulative dwell time exceeds the set warning threshold, a corresponding management response can be triggered.

[0112] For example, a petrochemical enterprise deployed multiple video surveillance systems in hazardous areas. Using a front-end algorithm, the system extracted data on the locations of dozens of workers during specific shifts. After processing the affinity between these locations, the system identified three primary locations, A, B, and C. The path A→B→C recurred within the constructed migration chain. Using a sliding window, the system extracted each path segment A→B→C and compared it with another segment A→B→C. The system found that their feature vectors were highly similar, and the spatial extents traversed by the paths were largely consistent, resulting in a repeatability score exceeding a set threshold. The system labeled this path segment as a periodic path and merged the areas it covered to form a repeating zone Z. Subsequent statistics revealed that person X had made five stops in zone Z, staying for 6, 8, 7, 5, and 9 minutes, respectively. The system calculated the total duration of this person's stay in zone Z as 35 minutes.

[0113] This implementation process can be applied not only to behavior supervision in dangerous areas, but also to repeated patrol identification, person-job matching assessment and abnormal behavior perception in multiple scenarios such as office parks and workshop work areas. Through the resident migration chain and repeated path analysis technology, it can provide the intelligent monitoring system with a deeper spatiotemporal behavior modeling capability, and improve the automation level of regional behavior understanding and risk warning. This method does not need to rely on external positioning devices, has strong environmental adaptability and engineering practicality, and can be embedded in existing video surveillance systems for low-cost deployment. Combined with the resident point detection technology and repeated path recognition mechanism provided by the present invention, it is possible to model and analyze human behavior in a non-invasive manner, which is a reliable, efficient and easy-to-integrate intelligent analysis method.

[0114] This embodiment enables accurate identification of repeated round-trip behavior and analysis of dwell patterns within the monitored area, effectively revealing potential behavioral characteristics such as abnormal patrols, illegal stops at designated locations, or frequent proximity. By constructing a dwell migration chain and introducing an affinity matrix to aggregate dwell points, the accuracy of regional divisions is improved while also enhancing the ability to express behavioral patterns across different time periods. Combining a comprehensive assessment of regional feature vector similarity and spatial location overlap, periodic path patterns can be efficiently identified and dwell time in relevant areas quantified, providing robust data support for identifying behavior in hazardous areas, assessing risk levels, and optimizing management strategies.

[0115] In an optional embodiment, determining whether the repeated travel area spatially overlaps with the dangerous equipment warning area, and if so, marking the repeated travel area as a suspicious area, and calculating the initial risk value based on the residence time includes:

[0116] Obtaining a set of boundary vertex coordinates of the reciprocating area and the dangerous equipment warning area, performing convex hull fitting on the set of boundary vertex coordinates to obtain area boundary lines, and calculating the horizontal and vertical projection ranges of the reciprocating area and the dangerous equipment warning area based on the area boundary lines;

[0117] Calculating the projected intersection area of ​​the repeated round-trip area and the dangerous equipment warning area based on the horizontal projection range and the vertical projection range, calculating the spatial overlap based on the ratio of the projected intersection area to the area of ​​the repeated round-trip area, and marking the repeated round-trip area with a spatial overlap exceeding a preset overlap threshold as a suspicious area;

[0118] Obtaining equipment type information, hazardous material information, and process parameter information of dangerous equipment, calculating the equipment hazard weight, extracting the residence time in the suspicious area, statistically analyzing the residence time by time period to obtain a residence time distribution sequence, calculating the residence probability based on the residence time distribution sequence, and calculating the time entropy value based on the residence probability;

[0119] The risk base value is calculated according to the spatial overlap, the residence time and the equipment hazard weight; the risk base value is weightedly combined with the time entropy value to obtain the risk initial value.

[0120] For example, a set of coordinates for the repeated round-trip area within the monitoring area is obtained, such as {(12.5, 35.2), (15.6, 38.7), (19.3, 37.9), (18.7, 33.4), (14.2, 32.6)}, as well as a set of coordinates for the dangerous equipment warning area, such as {(16.8, 34.5), (20.3, 36.2), (21.5, 33.8), (18.9, 31.7), (15.2, 32.3)}. A convex hull fit is performed on these coordinate points, and the convex hull algorithm (Graham scan method) is used to construct the area boundary. For the repeated round-trip area, the algorithm starts from the lowest point, sorts by polar angle, and then adds vertices in sequence, removing non-convex points, ultimately obtaining the repeated round-trip area boundary. Similarly, the dangerous equipment warning area boundary is obtained.

[0121] Calculate the projection range based on the obtained boundary lines. The horizontal projection of the reciprocating area is [12.5, 19.3], and the vertical projection is [32.6, 38.7]. The horizontal projection of the dangerous equipment warning area is [15.2, 21.5], and the vertical projection is [31.7, 36.2]. Calculate the intersection of the projections: the horizontal intersection is [15.2, 19.3], with a range length of 4.1; the vertical intersection is [32.6, 36.2], with a range length of 3.6. The area of ​​the projection intersection is calculated as 4.1 × 3.6 = 14.76 square units.

[0122] The area of ​​the repeated round-trip region was calculated using the polygon area calculation formula constructed from the boundary coordinates, resulting in an area of ​​approximately 29.42 square units. The spatial overlap is the ratio of the projected intersection area to the area of ​​the repeated round-trip region: 14.76 / 29.42 = 0.502. The preset overlap threshold is set to 0.35. Since 0.502 > 0.35, the system marks the repeated round-trip region as suspicious.

[0123] For suspicious areas, the system retrieves hazardous equipment information from the equipment database. For example, consider the equipment type as "autoclave," the hazardous material as "flammable chemicals," and the process parameters as "operating pressure 8.5 MPa, temperature 175°C." Based on the equipment type and the hazard rating table, the autoclave's base hazard level is 0.8. Combined with the flammable chemical's hazard factor of 0.9 and the process parameter's level of excess of 0.7, the calculated equipment hazard weight is 0.8 × 0.9 × 0.7 = 0.504.

[0124] The system extracts data on the length of time people spend in a suspicious area. For example, for seven consecutive days, the data shows the following: [25 minutes, 32 minutes, 28 minutes, 31 minutes, 45 minutes, 30 minutes, 26 minutes]. Based on the time period, the duration of stay is divided into five intervals: 0-15 minutes, 15-30 minutes, 30-45 minutes, 45-60 minutes, and over 60 minutes. The resulting distribution of stay times is [0, 4, 3, 0, 0], indicating that four days were spent between 15 and 30 minutes, and three days were spent between 30 and 45 minutes. The calculated probability of stay is [0, 0.571, 0.429, 0, 0].

[0125] The temporal entropy value is calculated based on the dwell probability, representing the uncertainty of the dwell time. A more uniform probability distribution indicates a higher temporal entropy value, indicating a more irregular behavior pattern. According to the temporal entropy calculation method, the temporal entropy value for this example is 0.684. The risk base value calculation comprehensively considers spatial overlap, dwell time, and equipment hazard weighting. A spatial overlap of 0.502 indicates that more than half of the areas overlap; the average dwell time is (25+32+28+31+45+30+26) / 7, which is approximately 31 minutes; and the equipment hazard weight is 0.504. The risk base value is calculated as 0.502 × 31 × 0.504 = 7.855. Finally, the risk base value and the temporal entropy value are weighted and combined to obtain the initial risk value. Assuming a temporal entropy weight coefficient of 0.3 and a risk base value weight of 0.7, the initial risk value is 7.855 × 0.7 + 0.684 × 0.3 = 5.703. This initial risk value will be used in subsequent risk level assessments and warning triggering decisions.

[0126] The above calculation results show that there is obvious spatial overlap between the repeated round-trip area and the dangerous equipment warning area. People appear in this area regularly and stay for a long time. The contact with high-risk equipment poses a potential safety risk, which requires the attention of safety management personnel and the implementation of corresponding preventive measures.

[0127] The system categorizes suspicious areas into different risk levels based on the initial risk value. For example, a value exceeding 5.5 is considered a high-risk area requiring immediate inspection; a value between 3.5 and 5.5 is considered a medium-risk area requiring regular inspections; and a value below 3.5 is considered a low-risk area requiring routine monitoring. In this example, a medium-risk value of 5.703 indicates a high-risk area, and the system will generate a high-priority security inspection task.

[0128] By obtaining the coordinates of the boundary vertices of the repeated round-trip area and the dangerous equipment warning area, the regional boundary line is constructed using convex hull fitting, thereby achieving accurate calculation of the horizontal and vertical projection ranges. The spatial overlap is calculated by the ratio of the projection intersection area to the regional area, effectively identifying suspicious areas with potential interaction risks. Unlike the existing technology that relies on rough bounding boxes or preset static area divisions, this application introduces a convex hull fitting method to improve the accuracy and adaptability of boundary identification and avoid the deviation of regular boundaries from modeling actual irregular spatial areas. Furthermore, by introducing multi-source data such as equipment type, hazardous materials, and process parameters, a hazard weight model is established. Combined with the distribution statistics of residence time and time entropy analysis, a dynamic risk perception mechanism for spatiotemporal interaction behavior is constructed. Existing technologies generally only judge risks based on residence time thresholds or spatial overlap, ignoring the temporal regularity of human behavior and the hazardous properties of the equipment itself. This application introduces the measurement method of residence probability and time entropy in time modeling, making risk calculation more dynamic and discriminative. Through the fusion analysis of spatial overlap, residence characteristics and equipment hazard level, high-risk areas can be identified more comprehensively and granularly, improving the accuracy of risk assessment and the foresight of early warning.

[0129] In an optional embodiment, extracting the action sequence of a person target in a suspicious area and performing spatiotemporal feature analysis to obtain a semantic description of the action, determining an out-of-norm action based on the semantic description of the action, and performing a first update on the initial risk value based on the duration of the out-of-norm action include:

[0130] Acquire a video sequence of a person in a suspicious area and extract the coordinate positions of the human skeleton nodes. Calculate the displacement distance and direction of the skeleton nodes between adjacent frames to obtain the node displacement vector. Establish a local polar coordinate system based on the spatial positions of different operating devices in the industrial scene, and map the node displacement vector to the corresponding polar coordinate system to obtain an action sequence that takes into account the device orientation.

[0131] Calculating the angle and distance between the action sequence and the device operating surface to obtain spatiotemporal features; extracting the interaction pattern between the person and the device based on the spatiotemporal features, and calculating the action continuity based on the temporal changes in the interaction pattern;

[0132] Calculating the fluctuation pattern of the action continuity and extracting semantic features; constructing an action semantic description considering device interaction based on the semantic features; matching the action semantic description with a standard action library and calculating a semantic difference value;

[0133] The semantic description of an action whose semantic difference value exceeds a preset threshold is marked as an out-of-specification action; the duration of the out-of-specification action is extracted; the risk update coefficient of the out-of-specification action is calculated based on the relative position relationship and interaction intensity between the out-of-specification action and the device, and the initial risk value is updated for the first time based on the risk update coefficient.

[0134] This invention provides a technical solution for extracting motion sequences of human targets within suspicious areas and performing risk assessment. In practice, a video surveillance system first acquires video sequence data of human targets within a designated suspicious area in an industrial environment. The system then uses a human pose estimation algorithm to extract the coordinates of key skeletal nodes from the video frames. This includes 2D or 3D coordinate information for 17 key points, including the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0135] For the extracted skeleton node sequence, calculate the displacement distance and displacement direction of each node between adjacent frames. For example, for the wrist node, calculate the displacement of the node from position (x t ,y t , z t ) to (x t+1 ,y t+1 , z t+1 ) to form a node displacement vector. For example, if an operator is in front of an electrical cabinet, the displacement vector of their right wrist node over five consecutive frames is [(3, 2, 1), (4, 3, 0), (5, 2, -1), (4, 1, -2), (3, 0, -1)] centimeters. This indicates that the hand approached the equipment and then retreated.

[0136] Taking into account the differences in the spatial orientation of equipment in industrial scenarios, a local polar coordinate system is established with the equipment operating surface as the reference. For example, for a vertically mounted control panel, a polar coordinate system is established with the center of the panel as the origin and the panel normal direction as the polar axis direction. The node displacement vector calculated above is mapped from the Cartesian coordinate system to the local polar coordinate system of the corresponding device to obtain a node motion representation that takes into account the orientation of the device. For example, the displacement vector of the operator's right wrist is expressed in the polar coordinate system of the electrical cabinet operating surface as [(2.5, 45°), (3.2, 30°), (4.1, 15°), (3.8, -10°), (2.2, -20°)], where the first value represents the radial distance (cm) and the second value represents the angle with the normal of the operating surface.

[0137] The interaction features between the action sequence and the device operating surface are further calculated, including distance changes and angle changes. For distance features, the time-varying sequence of vertical distances from key human nodes (such as the hand) to the device operating surface is calculated. For angle features, the sequence of angles between the node displacement vectors and the normal direction of the operating surface is calculated. In a specific implementation, the system recorded that the distance between the operator's right hand and the electrical cabinet panel changed as [50, 45, 40, 35, 30, 25, 20, 15, 10, 5] cm over 10 seconds, and the angle changed as [60°, 55°, 45°, 40°, 35°, 30°, 25°, 20°, 15°, 10°], indicating that the operator was gradually approaching and operating the device directly.

[0138] Based on these spatiotemporal features, the interaction patterns between the user and the device are extracted. Using a temporal pattern recognition algorithm, the system divides continuous distance and angle change sequences into meaningful interaction segments, such as the basic interaction unit of "approach-operate-leave." For each interaction segment, a motion continuity index is calculated, which reflects the smoothness of the changes in speed and direction of the movement. For example, during normal operation, the speed change curve of the hand movement should be relatively smooth, while abnormal operation may manifest as sudden acceleration or sudden changes in direction.

[0139] The system analyzes the fluctuation patterns of action continuity and extracts semantic features that can express action intent. A semantic dictionary for action is designed to map spatiotemporal feature patterns to predefined action semantic descriptions, such as "slow approach," "rapid withdrawal," "continuous operation," and "hesitation and wandering." In a real-world scenario, the system translates an operator's 10-second interaction with an electrical cabinet into a semantic sequence: "slow approach from the front (3 seconds) - hesitant pause (2 seconds) - light touch operation (1 second) - rapid withdrawal (4 seconds)."

[0140] The system maintains a library of standardized actions for different devices, containing standard semantic sequence templates for normal operation of each type of equipment. Extracted action semantic descriptions are matched against the standard actions in this library to calculate semantic difference values. This semantic difference calculation considers three dimensions: action type matching, temporal consistency, and execution standardization. For example, the standard semantic template for electrical cabinet operations is "slow approach from the front - smooth operation - slow retreat." The calculated difference value between this and the extracted semantic sequence is 0.35 (difference range 0-1).

[0141] When the calculated semantic difference value exceeds a preset threshold (e.g., 0.3), the system marks the action sequence as non-normative. The duration of the non-normative action is also recorded. For example, in the previous example, the "hesitation" action lasted for 2 seconds and was considered non-normative. The risk update coefficient is calculated based on the spatial location characteristics and interaction intensity of the non-normative action. Spatial location characteristics consider the relative position of the person and the device. For example, close proximity (less than the safe distance of 15 cm) or abnormal angles (lateral operation greater than 45 degrees) increase the risk factor. Interaction intensity considers the force and frequency of the action. For example, rapid and repeated touching of sensitive equipment increases the risk factor. In this example, the "hesitation" action occurred at 20 cm from the device, resulting in a low interaction intensity and a calculated risk update coefficient of 1.2.

[0142] The calculated risk update factor is used to perform a first update to the initial risk value. Assuming the initial risk value for this electrical cabinet operation is 50 (out of 100), the updated risk value is 50 x 1.2 = 60, indicating that the risk level has increased but has not yet reached the high-risk warning threshold (typically 75). This updated risk value will be used in subsequent risk assessments.

[0143] In this embodiment, by extracting video sequences of human targets within suspicious areas, obtaining the coordinate changes of human skeleton nodes, and introducing a local polar coordinate system under the premise of considering the spatial orientation of industrial equipment, the unified expression of action sequences in the equipment reference system is effectively achieved. Compared with the existing method of performing simple Euclidean distance analysis directly based on images or skeleton sequences, the adaptability and accuracy of action recognition in complex spatial scenarios are greatly improved. Furthermore, by calculating the angle and distance relationship between the action sequence and the equipment operating surface, the interaction characteristics between the person and the equipment are constructed, and the action continuity and semantic fluctuations are analyzed in combination with time series information, overcoming the defect of the existing technology that cannot accurately judge small non-standard operations. This application also introduces a matching mechanism between action semantic descriptions and standardized action libraries, which can identify non-standard operation behaviors at the semantic level, breaking through the problem of traditional methods lacking an explanatory basis for judging whether an action is standardized. At the same time, the risk assessment value is dynamically updated based on the continuity of non-standard actions and the intensity of spatial interaction, making the risk judgment more closely aligned with actual scene changes, with higher timeliness and practicality, thereby providing more fine-grained and targeted dynamic risk management capabilities in industrial safety monitoring.

[0144] In an optional embodiment, the type of work tools carried by the target person is identified and the identification result is checked against the work order. When an unregistered work tool is found, the risk value is updated a second time to obtain a final risk value. When the final risk value exceeds the warning threshold, alarm data is generated including:

[0145] Collecting tool instantaneous power data and operating current signals, analyzing tool load changes based on the instantaneous power data, calculating tool start and stop frequencies based on the operating current signals, and combining the load changes and start and stop frequencies to determine tool usage status;

[0146] Acquire an image of the working area, extract the geometric dimensions and appearance features of the tool from the image of the working area, combine the use status of the tool with the geometric dimensions and appearance features, and identify the type of the tool;

[0147] Obtaining work order records, comparing the tool type with the work order, and when unregistered tools are detected, calculating the tool hazard level based on the tool's power change trend and usage frequency, and determining the distance constraint between tools based on the location distribution of the tools in the work area;

[0148] Obtaining the combustible gas concentration and ambient temperature of the operating area, determining whether there is a fire hazard when using the tool based on the combustible gas concentration and ambient temperature, and updating the last updated risk value a second time based on the determination result and the tool hazard level to obtain a final risk value;

[0149] A dynamic warning threshold is calculated based on the real-time change trend of personnel density, equipment operating status, and environmental parameters in the operating area. When the final risk value exceeds the dynamic warning threshold, alarm data containing tool information is generated.

[0150] For example, the power data is first sampled, such as collecting data every 0.5 seconds, and the power change rate is calculated: the power values ​​at adjacent time points are compared. If the change exceeds a preset threshold, such as 20%, it is determined that the load has changed significantly. At the same time, the system calculates the start and stop frequency of the tool based on the operating current signal. By setting a current threshold, such as 0.5 amperes, when the current rises from below the threshold to above the threshold, it is recorded as a start, and vice versa. It is recorded as a stop. For example, for an electric drill tool, 10 starts and 10 stops are detected within 30 minutes, which means the start and stop frequency is determined to be 20 times. The system combines the load change and the start and stop frequency to judge the tool usage status. For example, when a cutting machine is in use, if the power is detected to fluctuate from 500W to 1200W, and the start and stop frequency is 15 times within 30 minutes, it is determined to be "intermittent high load usage state".

[0151] The system captures an image of the work area and uses image processing to extract the tool's geometric dimensions and appearance features. For geometric dimension extraction, the system uses a pixel-to-actual-size conversion method. For example, at a standard distance, 100 pixels corresponds to 10 centimeters. The system detects the tool's outline and calculates its length, width, and height. For example, a welder is detected to be 45 centimeters long, 30 centimeters wide, and 25 centimeters high. For appearance feature extraction, the system analyzes the tool's color distribution, shape, and texture features, such as identifying the tooth structure of a power saw or the spiral drill bit of an electric drill. The system combines the tool's usage status with its geometric dimensions and appearance features to identify the tool type using a feature matching algorithm. For example, if a tool approximately 30 centimeters long and 10 centimeters wide, with a cylindrical shape and a spiral structure at the front, is detected, and its usage is characterized by short, high-power usage, the system identifies it as an electric drill.

[0152] The system retrieves the work order record and compares the identified tool type with the work order. The work order contains a list of permitted tools, such as drills, cutters, and measuring instruments. If the system detects a tool not registered on the work order, such as a welder that is not registered on the work order, it calculates the tool's hazard level based on the tool's power fluctuation trend and usage frequency. The hazard level is categorized as low, medium, or high. The calculation method is as follows: if the power fluctuation trend is stable (fluctuation less than 30%) and the usage frequency is low (less than 5 starts and stops within 30 minutes), the hazard level is low; if the power fluctuation trend is volatile (fluctuation between 30% and 60%) or the usage frequency is moderate (5 to 15 starts and stops within 30 minutes), the hazard level is medium; if the power fluctuation trend is drastic (fluctuation greater than 60%) or the usage frequency is high (starts and stops greater than 15 times within 30 minutes), the hazard level is high. For example, if an unregistered welder has a power fluctuation of 70% and starts and stops 12 times within 30 minutes, the system will determine its hazard level as high.

[0153] The system determines the distance constraints between tools based on their distribution within the work area. By analyzing the area image, the system calculates the distance between each tool and sets a minimum safe distance threshold. For high-risk tools like welders, the minimum safe distance from other tools is 3 meters; for medium-risk tools like cutters, the minimum safe distance is 2 meters; and for low-risk tools like drills, the minimum safe distance is 1 meter. If the actual distance is less than the safe distance, the system increases the risk level assessment.

[0154] The system collects combustible gas concentration and ambient temperature data in the work area to determine whether the use of tools poses a fire hazard. Combustible gas concentrations, such as methane and propane concentrations, are measured using gas sensors, while ambient temperature is measured using temperature sensors. Combustible gas concentration thresholds are set, such as a methane concentration threshold of 0.5% by volume in air and an ambient temperature threshold of 40°C. When the combustible gas concentration or ambient temperature exceeds the threshold, the system determines that a fire hazard exists. Based on the assessment results and the tool's hazard level, the system performs a second update of the risk value. The update is as follows: if an unregistered tool is detected and the hazard level is low, the risk value is increased by 10%; if the hazard level is medium, the risk value is increased by 30%; and if the hazard level is high, the risk value is increased by 50%. If a fire hazard also exists, the risk value is increased by an additional 20%. For example, if the initial risk value is 50 and a high-risk, unregistered tool, such as a welding machine, is detected and presents a fire hazard, the final risk value is updated to 50 × (1 + 50% + 20%) = 85.

[0155] Dynamic warning thresholds are calculated based on real-time trends in the work area's occupancy density, equipment operating status, and environmental parameters. Occupancy density is calculated by dividing the number of people in the area by the area; equipment operating status includes factors such as operating hours and load levels; and environmental parameters include temperature, humidity, and gas concentrations. The system sets a baseline threshold based on these parameters, such as 70 points, and dynamically adjusts based on parameter changes: for every 0.1 person / square meter increase in occupancy density, the threshold decreases by 5 points; for every hour of continuous equipment operation time, the threshold decreases by 3 points; and for every 5°C increase in ambient temperature, the threshold decreases by 2 points. For example, if the initial threshold is 70 points and the occupancy density is 0.3 people / square meter, the equipment has been operating continuously for 2 hours, and the ambient temperature is 10°C above the baseline, the dynamic warning threshold is adjusted to 70 - 5 × 3 - 3 × 2 - 2 × 2 = 53 points.

[0156] When the final risk value exceeds the dynamic warning threshold, the system generates alarm data, including information about detected unregistered tools, tool hazard levels, usage status, location information, and environmental parameter data, to provide decision support for safety managers.

[0157] This technical solution achieves precise identification and risk assessment of tool usage status through multi-dimensional data fusion, significantly enhancing the intelligent level of safety management in petroleum and petrochemical sites. First, by combining the tool's instantaneous power and operating current signals, it dynamically determines the tool's start-stop frequency and load changes, ensuring real-time monitoring of tool usage status and addressing the misjudgment caused by traditional monitoring methods that rely solely on a single signal. Second, by extracting the tool's geometric dimensions and appearance characteristics from images of the work area, this visual information is combined with electrical signals to accurately identify the tool type, which is more efficient and accurate than traditional methods based on a single signal or manual verification. Identification results are then compared with work orders to promptly identify unregistered tools, enhancing operational compliance management. Furthermore, tool hazard level assessment and location distribution distance constraints are introduced to enhance the proactive assessment of potential safety hazards. Combustible gas concentration and ambient temperature are combined to scientifically assess the fire risk associated with tool usage, addressing the lack of environmental considerations in existing technologies. Finally, warning thresholds are dynamically calculated and adjusted in real time based on personnel density, equipment operating status, and environmental parameter changes. This improves the sensitivity and accuracy of warnings, ensuring that alarm data containing detailed tool information is generated promptly when risks exceed limits, effectively preventing safety incidents. Compared to traditional technologies, this overall solution achieves more comprehensive, dynamic, and accurate risk assessment and warning.

[0158] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including:

[0159] processor;

[0160] a memory for storing processor-executable instructions;

[0161] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0162] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0163] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying personnel safety violations in petroleum and petrochemical scenes based on video analysis, characterized by: include: Obtain video surveillance data around dangerous equipment in the petrochemical area and perform target detection on the video surveillance data to obtain human targets; A speed-time curve is established based on the position changes of the human target in consecutive video frames. The dwell points are obtained by detecting the trough points of the speed-time curve. A dwell migration chain is constructed based on the temporal relationship of the dwell points. The periodic path segments in the dwell migration chain are detected, and the areas corresponding to the periodic path segments are determined as repeated round-trip areas. The dwell duration of the repeated round-trip areas is calculated. Determine whether the repeated travel area overlaps with the dangerous equipment warning area. If there is overlap, mark the repeated travel area as a suspicious area and calculate the initial risk value based on the residence time. Extract the action sequence of human targets in the suspicious area and perform spatiotemporal feature analysis to obtain the action semantic description. According to the action semantic description, the non-standard actions are judged, and the initial risk value is updated for the first time based on the duration of the non-standard actions. Identify the type of work tools carried by the target personnel and check the identification results with the work order. When unregistered work tools appear, the risk value is updated for the second time to obtain the final risk value. When the final risk value exceeds the warning threshold, alarm data is generated.

2. The method according to claim 1, characterized in that Obtain video surveillance data around dangerous equipment in the petrochemical area, and perform target detection on the video surveillance data to obtain human targets including: Obtain video surveillance data around dangerous equipment in the petrochemical area and perform frame segmentation to obtain images to be processed; Performing motion foreground analysis on the image to be processed and obtaining a moving target region through background modeling; performing adaptive threshold segmentation on the moving target region to obtain a motion foreground mask image; and extracting a target candidate region from the image to be processed based on the motion foreground mask image; Extracting human body feature parameters from the target candidate area, including height ratio, limb length ratio, and contour shape features; calculating a target similarity score based on the human body feature parameters, and screening to obtain the human body target area; Extracting the spatiotemporal position information of the human target area, including the image coordinate position and the area range; establishing a target tracker, performing temporal association on the human targets in consecutive frames, and outputting the human targets including the target number and position information.

3. The method according to claim 1, characterized in that The speed-time curve is established based on the position change of the human target in the continuous video frames. The dwell point is obtained by detecting the trough point of the speed-time curve. The following steps are included: Obtain the position coordinates of the human target in continuous video frames, and calculate the displacement components of adjacent position coordinates in the horizontal and vertical directions respectively; calculate the displacement distance and displacement direction based on the obtained horizontal displacement component and vertical displacement component to obtain the displacement vector; Calculating the speed and motion direction according to the displacement vector and the video frame rate; arranging the speed and motion direction in chronological order to construct an initial motion feature sequence; Calculating the mean and standard deviation of the velocity magnitudes in the initial motion feature sequence, determining a filter window base size, and simultaneously calculating the local fluctuation intensity of the velocity magnitudes, dynamically adjusting the filter window base size to obtain an adaptive window size; performing a sliding average filter on the velocity magnitudes using the adaptive window size to obtain a smoothed velocity sequence; and constructing a velocity-time curve based on the smoothed velocity sequence; Calculating a forward difference value and a backward difference value of the smoothed velocity sequence, and identifying a velocity trough position according to a change in the signs of the forward difference value and the backward difference value; The motion direction at the velocity trough position is extracted, and the angle between adjacent motion directions is calculated to obtain the motion direction change value. Based on the speed value, speed duration and motion direction change value at the velocity trough position, the trough point that meets the static state judgment conditions is screened out as the residence point.

4. The method according to claim 1, wherein The dwell migration chain is constructed based on the temporal relationship of the dwell points. The periodic path segments in the dwell migration chain are detected. The areas corresponding to the periodic path segments are determined as repeated round-trip areas. The dwell time in the repeated round-trip areas is calculated. Extracting the spatiotemporal information of the dwell points, calculating the Mahalanobis distance between the dwell points to obtain spatial proximity; calculating the time intervals of the dwell points to obtain temporal correlation; and constructing a dwell point affinity matrix based on the spatial proximity and temporal correlation. Performing region growing on the resident points based on the affinity matrix to obtain a resident region; extracting a regional feature vector of the resident region, and constructing a resident migration chain according to a temporal relationship of the resident region; Extracting resident migration segments from the resident migration chain using a variable-length sliding window, calculating the similarity of regional feature vectors and the regional position overlap between adjacent resident migration segments, calculating the repeatability score of the resident migration segments based on the regional feature vector similarity and the regional position overlap, and marking the resident migration segments with a repeatability score higher than a preset repeatability score threshold as periodic path segments; The spatial areas covered by the periodic path segments are merged into a repeated round-trip area, the entry time and the exit time of the stay points in the repeated round-trip area are extracted, the single stay duration of the stay points is calculated, and the single stay durations are accumulated to obtain the total stay duration of the repeated round-trip area.

5. The method according to claim 1, wherein Determine whether the repeated travel area overlaps with the dangerous equipment warning area. If there is overlap, mark the repeated travel area as a suspicious area and calculate the initial risk value based on the residence time, including: Obtaining a set of boundary vertex coordinates of the reciprocating area and the dangerous equipment warning area, performing convex hull fitting on the set of boundary vertex coordinates to obtain area boundary lines, and calculating the horizontal and vertical projection ranges of the reciprocating area and the dangerous equipment warning area based on the area boundary lines; Calculating the projected intersection area of ​​the repeated round-trip area and the dangerous equipment warning area based on the horizontal projection range and the vertical projection range, calculating the spatial overlap based on the ratio of the projected intersection area to the area of ​​the repeated round-trip area, and marking the repeated round-trip area with a spatial overlap exceeding a preset overlap threshold as a suspicious area; Obtaining equipment type information, hazardous material information, and process parameter information of dangerous equipment, calculating the equipment hazard weight, extracting the residence time in the suspicious area, statistically analyzing the residence time by time period to obtain a residence time distribution sequence, calculating the residence probability based on the residence time distribution sequence, and calculating the time entropy value based on the residence probability; The risk base value is calculated according to the spatial overlap, the residence time and the equipment hazard weight; the risk base value is weightedly combined with the time entropy value to obtain the risk initial value.

6. The method according to claim 1, characterized in that Extract the action sequence of human targets in the suspicious area and perform spatiotemporal feature analysis to obtain the action semantic description. According to the action semantic description, determine the non-standard action. The first update of the initial risk value is based on the duration of the non-standard action. The following steps are included: Acquire a video sequence of a person in a suspicious area and extract the coordinate positions of the human skeleton nodes. Calculate the displacement distance and direction of the skeleton nodes between adjacent frames to obtain the node displacement vector. Establish a local polar coordinate system based on the spatial positions of different operating devices in the industrial scene, and map the node displacement vector to the corresponding polar coordinate system to obtain an action sequence that takes into account the device orientation. Calculating the angle and distance between the action sequence and the device operating surface to obtain spatiotemporal features; extracting the interaction pattern between the person and the device based on the spatiotemporal features, and calculating the action continuity based on the temporal changes in the interaction pattern; Calculating the fluctuation pattern of the action continuity and extracting semantic features; constructing an action semantic description considering device interaction based on the semantic features; matching the action semantic description with a standard action library and calculating a semantic difference value; The semantic description of an action whose semantic difference value exceeds a preset threshold is marked as an out-of-specification action; the duration of the out-of-specification action is extracted; the risk update coefficient of the out-of-specification action is calculated based on the relative position relationship and interaction intensity between the out-of-specification action and the device, and the initial risk value is updated for the first time based on the risk update coefficient.

7. The method according to claim 1, characterized in that Identify the type of work tools carried by the target personnel and check the identification results with the work order. If an unregistered work tool appears, the risk value is updated a second time to obtain the final risk value. When the final risk value exceeds the warning threshold, alarm data is generated including: Collecting tool instantaneous power data and operating current signals, analyzing tool load changes based on the instantaneous power data, calculating tool start and stop frequencies based on the operating current signals, and combining the load changes and start and stop frequencies to determine tool usage status; Acquire an image of the working area, extract the geometric dimensions and appearance features of the tool from the image of the working area, combine the use status of the tool with the geometric dimensions and appearance features, and identify the type of the tool; Obtaining work order records, comparing the tool type with the work order, and when unregistered tools are detected, calculating the tool hazard level based on the tool's power change trend and usage frequency, and determining the distance constraint between tools based on the location distribution of the tools in the work area; Obtaining the combustible gas concentration and ambient temperature of the operating area, determining whether there is a fire hazard when using the tool based on the combustible gas concentration and ambient temperature, and updating the last updated risk value a second time based on the determination result and the tool hazard level to obtain a final risk value; A dynamic warning threshold is calculated based on the real-time change trend of personnel density, equipment operating status, and environmental parameters in the operating area. When the final risk value exceeds the dynamic warning threshold, alarm data containing tool information is generated.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images