Methods for Identifying and Detecting Safety Protection Equipment for High-Risk Operation Videos in Power Plants
By combining image preprocessing and the YOLOv8 target detection algorithm with continuous frame analysis, the problems of false alarms and missed alarms in the detection of safety protection equipment in high-risk operation videos of power plants have been solved, achieving higher detection accuracy and reliability.
Patent Information
- Application Number
- CN202611242558.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-10
AI Technical Summary
In high-risk operation videos of power plants, existing technologies are prone to affecting the detection of safety protection equipment due to factors such as changes in lighting, metal reflections, dust interference, overlapping of personnel, and equipment obstruction. This can lead to false alarms, missed alarms, and invalid records, making it difficult to accurately identify the equipment wearing status.
By employing image preprocessing and target detection algorithms, including bilateral filtering, desharpening, adaptive histogram equalization, and gamma correction, combined with the YOLOv8 target detection algorithm, the wearing status of safety protective equipment is identified and confirmed. False alarms and false negatives are reduced through continuous frame analysis and position constraints.
It improves the accuracy, stability, and traceability of safety equipment wearing detection in complex scenarios, reduces false alarms and false negatives, and enhances the reliability of detection.
Smart Images

Figure CN123071933A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of power production safety monitoring and computer vision recognition technology, and more specifically, to a method for identifying and detecting safety protection equipment for high-risk operation videos in power plants. Background Technology
[0002] High-risk operations such as confined space maintenance in power plants typically involve capturing video footage using camera equipment and employing object detection algorithms to identify personnel and their safety helmets, safety belts, and protective clothing to determine the wearing status of safety protective equipment. Current technologies often rely on single-frame detection results, confidence thresholds, detection frame area, or the relative position of equipment to make judgments. However, work sites are frequently accompanied by changes in lighting, metallic reflections, dust interference, overlapping personnel, equipment obstruction, and image boundary truncation, which can easily cause equipment detection frames to temporarily disappear, change in scale, shift in position, or be incorrectly assigned. Directly identifying these anomalies as a lack of equipment can easily lead to false alarms; relaxing the judgment criteria may miss truly missing behaviors. The conclusions regarding equipment wearing in adjacent image frames may change repeatedly, leading to switching of verification targets and the creation of invalid records.
[0003] To address the above problems, this invention proposes a solution. Summary of the Invention
[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method for identifying and detecting safety protection equipment for high-risk operation videos in power plants, so as to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: In a preferred embodiment, it includes: Obtain high-risk operation videos and the categories of required safety protective equipment, and perform image preprocessing on the high-risk operation videos to obtain preprocessed image frames; Target detection is performed on the preprocessed image frames to obtain personnel detection boxes and safety protection equipment detection boxes; Based on the position of the safety protection equipment detection frame relative to the personnel detection frame, the safety protection equipment detection frame and the personnel detection frame are matched to obtain the corresponding results; Based on the corresponding results and the required category of safety protective equipment, determine the safety protective equipment wearing result for each personnel detection frame, and generate a detection record of non-compliance with safety protective equipment wearing requirements based on the continuous safety protective equipment wearing results.
[0006] In a preferred embodiment, the step of performing image preprocessing on the high-risk operation video to obtain preprocessed image frames includes: Determine the boundary of the work area in each image frame of the high-risk operation video, retain the pixels within the boundary of the work area, and obtain the work area image; The image of the work area is sequentially subjected to bilateral filtering and desharpening to obtain an edge-enhanced image; The edge-enhanced image is sequentially subjected to contrast-limited adaptive histogram equalization and gamma correction to obtain the gamma-corrected image. Based on the proportion of bright pixels in the gamma-corrected image, the pixels exceeding the brightness threshold are compressed to obtain the preprocessed image frame.
[0007] In a preferred embodiment, the step of performing target detection on the preprocessed image frame to obtain personnel detection boxes and safety protection equipment detection boxes includes: The preprocessed image frame is input into the trained YOLOv8 object detection algorithm to obtain candidate detection boxes with category and detection confidence. Delete candidate detection boxes whose detection confidence is lower than the detection confidence threshold of the corresponding category; For multiple candidate detection boxes of the same category that overlap with each other, the candidate detection boxes with higher detection confidence are retained according to the overlap ratio between the candidate detection boxes. Candidate detection boxes categorized as "personnel" are identified as the "personnel detection boxes," and candidate detection boxes categorized as "safety protection equipment" are identified as the "safety protection equipment detection boxes."
[0008] In a preferred embodiment, the color of the work vest corresponding to each personnel role is determined according to the personnel role identification regulations performed on site; The image within each personnel detection frame is converted into three color components: hue, saturation, and brightness. Based on the preset hue range, saturation lower limit, and brightness range, the work vest pixels corresponding to each work vest color are extracted. The work vest pixels are subjected to erosion and dilation processing and connected region filtering to obtain the work vest region corresponding to each work vest color, and the proportion of the number of pixels in each work vest region to the number of pixels in the personnel detection box is calculated. The ratios are compared with preset work vest area ratio thresholds, and the personnel role corresponding to the personnel detection box is determined based on the comparison results.
[0009] In a preferred embodiment, the step of associating the safety protection equipment detection frame with the personnel detection frame based on the position of the safety protection equipment detection frame relative to the personnel detection frame includes: Based on the position of various safety protection equipment relative to personnel in the marked images, determine the location range of each type of safety protection equipment; Calculate the lateral and longitudinal relative positions of the center point of each of the safety protection equipment detection frames within each of the personnel detection frames; A safety protection equipment detection frame whose center point is located within the personnel detection frame, and whose horizontal and vertical relative positions are within the location range of the corresponding category of safety protection equipment, corresponds to the personnel detection frame. When the same safety protection equipment detection frame meets the corresponding conditions of multiple personnel detection frames, the ratio of the intersection area between the safety protection equipment detection frame and each of the personnel detection frames is calculated, and it is matched with the personnel detection frame with the largest intersection area ratio. When the intersection area ratios are the same, it is matched with the personnel detection frame with the closest center point.
[0010] In a preferred embodiment, before determining the safety protection equipment wearing result corresponding to each of the personnel detection frames, the method further includes: The review observation segments are obtained from consecutive preprocessed image frames in chronological order, and the personnel detection boxes in adjacent preprocessed image frames are temporarily matched to form a temporary personnel sequence. For each type of safety protection equipment required to be worn in the temporary personnel sequence, a reference image frame is selected from the review observation segment where the personnel detection frame does not touch the image boundary, the overlap ratio with other personnel detection frames is not greater than the preset personnel overlap reference value, and the safety protection equipment detection frame belongs to a unique entity. Based on the detection confidence of the safety protection equipment detection frame in the reference image frame, the ratio of the detection frame area to the corresponding personnel detection frame area, and the relative position of the center point, the confidence change interval, scale change interval, and position activity interval are determined, and the confidence change interval, scale change interval, and position activity interval are updated according to the newly added reference image frame. Determine the following relationship between the current safety protection equipment detection frame and the corresponding personnel detection frame, as well as the imaging state of the current preprocessed image frame. Based on the following relationship, imaging state, and comparison results between the current safety protection equipment detection frame and the confidence change interval, scale change interval, and position activity interval, determine the review status and generate a review object based on the review status.
[0011] In a preferred embodiment, determining the tracking relationship between the current safety protection equipment detection frame and the corresponding personnel detection frame includes: Calculate the lateral and longitudinal movement distances of the center points of the safety protection equipment detection frames and the corresponding personnel detection frames in adjacent preprocessed image frames, respectively. Subtract the lateral and longitudinal movement distances of the center point of the personnel detection frame from the lateral and longitudinal movement distances of the center point of the safety protection equipment detection frame to obtain the lateral follow-up difference and the longitudinal follow-up difference. Then, normalize the lateral follow-up difference and the longitudinal follow-up difference according to the width and height of the previous personnel detection frame. The median values of the lateral and longitudinal motion differences within the reviewed observation segment are obtained respectively, and the deviation of the current lateral and longitudinal motion differences from the corresponding median values is calculated. When the deviation is within the allowable range of lateral and longitudinal movement respectively, and the safety protection equipment detection frame does not change position and does not continuously reach the number of confirmation frames in the opposite direction of movement, the safety protection equipment detection frame is determined as a detection result that moves with the personnel; otherwise, it is determined as a non-following detection result.
[0012] In a preferred embodiment, determining the imaging state of the current preprocessed image frame includes: Calculate the average brightness difference and the change in the proportion of bright pixels within the corresponding person detection boxes in adjacent preprocessed image frames, and calculate the change in the standard deviation of pixel brightness in a predetermined width region around the person detection box; Based on the average brightness difference, the change in the proportion of bright pixels, and the change in the standard deviation of pixel brightness in the reference image frame, the corresponding reference change range is determined respectively; when any change exceeds the corresponding reference change range, the current preprocessed image frame is determined as the imaging state change frame; when all changes in a consecutive preset number of preprocessed image frames are within the corresponding reference change range, the corresponding preprocessed image frame is determined as the imaging state stable frame. An anomaly is defined as the first time the detection confidence level falls below the lower boundary of the confidence change interval, the first time the ratio of the detection frame area to the corresponding personnel detection frame area falls below the lower boundary of the scale change interval, or the first time the safety protection equipment detection frame partially disappears. When the difference in frame number between the detected anomaly and the frame of the change in imaging state is not greater than the number of synchronous confirmation frames, and the detection result of the safety protection equipment recovers to the corresponding change range in the subsequent frame of stable imaging state, the detected anomaly is determined to be a temporary imaging fluctuation; otherwise, the detected anomaly is determined to be a persistent anomaly.
[0013] In a preferred embodiment, determining the review status and generating a review object based on the review status includes: Based on the location range of the corresponding category of safety protection equipment, an area where the equipment should appear is established within each of the personnel detection frames. The portion of the area where the equipment should appear that is not covered by other personnel detection frames and does not exceed the boundary of the preprocessed image frame is determined as an effective visible area. Based on the proportion of the effective visible area to the area where the equipment should appear, it is determined whether the area where the equipment should appear is in a visible state or a restricted state. When the required safety protective equipment is not detected, if the area where the equipment should appear is in a restricted state, the verification observation segment is extended, and if it is still in a restricted state after reaching the maximum observation length, it is determined that the visibility condition is insufficient; if the area where the equipment should appear is in a visible state and the corresponding preprocessed image frame is an imaging state stable frame, it is determined that the stable state is missing when the number of consecutive undetected image frames reaches the current missing confirmation length. When the same safety protection equipment detection frame meets the corresponding conditions of multiple personnel detection frames, the corresponding relationship is re-determined based on the intersection area ratio and follow-up relationship between the safety protection equipment detection frame and each personnel detection frame. If the corresponding relationship cannot be uniquely determined, it is determined to be a conflict of ownership. Based on persistent anomalies, location anomalies, insufficient visibility conditions, missing stable states, or ownership conflicts, corresponding verification image regions are formed. These verification image regions are then input into the Faster R-CNN object detection algorithm, and the verification object is generated based on the verification detection results.
[0014] The technical effects and advantages of this invention's method for identifying and detecting safety protection equipment in high-risk power plant operation videos are as follows: This invention improves the recognizability of target features by defining the working area and processing images under low light, strong light, reflective light, and dust interference. It reduces equipment mismatch between personnel through personnel role recognition, equipment position constraints, and attribution determination. By analyzing the confidence changes, scale changes, positional activity, follow-up relationships, imaging status, and effective visibility conditions of the detection frame using continuous frame analysis, it can distinguish between temporary imaging fluctuations, persistent anomalies, missing stable states, and attribution conflicts. It also performs re-inspection of difficult areas, thereby reducing false alarms, missed alarms, and invalid records, and improving the accuracy, stability, and traceability of equipment wearing detection in complex scenarios. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the overall process of the safety protection equipment identification and detection method for high-risk operation videos in power plants, based on the present invention.
[0016] Figure 2 This invention presents a flowchart of the image preprocessing process for a safety protection equipment identification and detection method for high-risk operation videos in power plants. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] This invention discloses a method for identifying and detecting safety protection equipment in high-risk operation videos at power plants, comprising: like Figure 1 As shown, in step one, the work number, work type, work area, start time, end time, and required safety equipment category for this high-risk operation are retrieved from the approved work order, work plan, or safety operation record. If actual start and end times exist, they are used; otherwise, the approved planned start and end times are used. The required safety equipment category is determined according to the safety measures recorded in the work order or work plan. If the work order or work plan does not record each item, the currently executed safety operation procedure is queried according to the work type, and the corresponding safety helmet, safety belt, protective clothing, and safety equipment category are retrieved.
[0019] The system determines which camera can capture footage of the work area by querying the corresponding records between the camera equipment and the work area. These records are established during camera equipment installation or debugging and include the camera equipment number, installation location, shooting direction, and the area covered by the camera's view. When the camera's shooting direction is fixed, the video generated by that camera is read. When the camera can rotate, it is adjusted to a preset position corresponding to the work area, and the video generated at that preset position is then read. This preset position is the camera's rotation angle and zoom state saved during debugging, after the work area is fully aligned with the camera's view, and is stored in correspondence with the camera equipment number and the work area.
[0020] Video is retrieved from the designated camera equipment or an existing on-site video management platform, and high-risk operation videos are extracted according to the start and end times. During extraction, the original timestamp, camera equipment number, image width, image height, and frame rate of the video are preserved. The timestamp indicates the actual capture time of the image, the camera equipment number indicates the image source, the image width and height are used to determine the coordinates of the operation area in the image, and the frame rate is used to maintain the time interval between adjacent image frames.
[0021] Decode the high-risk operation video according to its original frame order to obtain continuous original image frames. Maintain the aspect ratio of the original image frames during decoding; do not stretch the images. Assign frame numbers according to the chronological order of the original image frames in the high-risk operation video, and store each original image frame with its corresponding timestamp, camera device number, and frame number.
[0022] The work area is determined from the original image frame. If an electronic fence has been set up on site, the coordinates of the boundary points of the electronic fence in the camera frame are read. If no electronic fence has been set up on site, an original image frame is selected from the same camera equipment, the same shooting direction, and the same zoom state. At least three non-collinear boundary points are marked along the boundary of the actual work area in a clockwise or counterclockwise direction. Adjacent boundary points are connected in sequence, and the last boundary point is connected to the first boundary point to form a closed polygon.
[0023] The horizontal and vertical scale coordinates of each boundary point are obtained by dividing the x-coordinate of each boundary point by the image width of the original image frame and dividing the y-coordinate of each boundary point by the image height of the original image frame. These scale coordinates are then stored in relation to the camera device number, shooting direction, and zoom status. When the image width or image height of the high-risk operation video changes, the horizontal scale coordinates are multiplied by the changed image width, and the vertical scale coordinates are multiplied by the changed image height to obtain the changed boundary point coordinates.
[0024] Check whether the line segments formed by adjacent boundary points intersect, and check whether each boundary point is within the coordinate range of the original image frame. If the line segments do not intersect and all boundary points are within the coordinate range, use the closed polygon as the working area; if the line segments intersect or there are boundary points outside the coordinate range, record the corresponding camera device number and frame number, and stop processing the original image frame.
[0025] Each original image frame is processed according to a closed polygon, retaining pixels inside and at the boundaries of the closed polygon, while setting the luminance and color components of pixels outside the closed polygon to zero, thus obtaining the work area image. The image width, image height, and origin of the work area image are kept consistent with the original image frame, ensuring that the pixel coordinates in the work area image directly correspond to the pixel coordinates in the original image frame. Figure 2 As shown.
[0026] Images for parameter determination are selected from historical high-risk operation videos captured by the same camera equipment, from the same shooting direction, and under the same zoom setting. These images include those with normal lighting, low illumination, dust interference, strong light, and equipment surface reflections. Personnel outlines, safety helmet outlines, safety belt outlines, protective clothing outlines, and background areas excluding personnel and safety equipment are marked on these images. Personnel and safety equipment outlines are used to examine edge position changes before and after image processing, while the background area is used to examine noise changes before and after image processing.
[0027] Bilateral filtering is performed on the image of the working area. The neighborhood range, spatial distance parameter, and pixel difference parameter used in bilateral filtering are determined by the image based on the parameters. The neighborhood range represents the range of surrounding pixels participating in the filtering with the current pixel as the center. The spatial distance parameter is used to reduce the influence of pixels farther away from the current pixel on the filtering result. The pixel difference parameter is used to reduce the influence of pixels with large brightness or color differences on the filtering result.
[0028] Multiple sets of different neighborhood range, spatial distance parameters, and pixel difference parameters are set, and bilateral filtering is performed on the image for each parameter setting. For each set of parameters, the standard deviation of pixel brightness in the filtered background region is calculated, and the average distance between the filtered personnel contour and each safety equipment contour and the manually marked contour is calculated. The average distance is obtained by taking the shortest distance from each pixel on the filtered contour to the manually marked contour and then calculating the average of all shortest distances.
[0029] The standard deviation of background region pixel brightness and the average contour distance corresponding to each set of parameters are normalized. During normalization, the minimum value of the same index is converted to zero, the maximum value is converted to one, and the remaining values are converted to values between zero and one according to their position between the minimum and maximum values. The normalized standard deviation of background region pixel brightness and the normalized average contour distance are added together, and the set of parameters with the smallest sum is selected as the bilateral filter parameters. The selected bilateral filter parameters are used to process the image of the working area to obtain the denoised image.
[0030] Specifically, during bilateral filtering, each pixel in the working area image is taken as the current pixel, and surrounding pixels are read within the neighborhood centered on the current pixel. The spatial distance influence value is determined based on the pixel coordinate distance between the current pixel and surrounding pixels, and the pixel difference influence value is determined based on the difference in luminance or color components between the current pixel and surrounding pixels. The product of the spatial distance influence value and the pixel difference influence value is used as the filtering weight corresponding to that surrounding pixel. The luminance or color component of each surrounding pixel within the neighborhood is multiplied by its corresponding filtering weight, summed, and then divided by the sum of all filtering weights to obtain the filtered luminance or color component of the current pixel.
[0031] When the work area image is a color image, bilateral filtering can be performed on the red, green, and blue color components separately. Alternatively, the work area image can be first converted into an image representation containing both luminance and color components, and then bilateral filtering can be performed on at least the luminance component. Using the latter method, the correspondence between the original color components and the filtered luminance components is preserved after filtering.
[0032] When obtaining the outlines of personnel and various safety protection equipment from the filtered image, the brightness change of a predetermined width region on both sides of the manually marked outline can be calculated, and pixels whose brightness change reaches the edge judgment value are identified as the pixels of the filtered outline. The edge judgment value is determined by the image through parameter determination, selecting the edge judgment value that minimizes the average distance between the filtered outline and the manually marked outline.
[0033] When performing normalization, for any value within the same indicator, the normalized value is obtained by subtracting the minimum value of the same indicator from that value, and then dividing by the difference between the maximum and minimum values of the same indicator. When the maximum and minimum values of the same indicator are equal, all normalized values corresponding to that indicator are set to zero to avoid division by zero.
[0034] The denoised image is desharpened. First, a Gaussian filter is used to smooth the denoised image to obtain a smoothed image. Then, the brightness value of each pixel in the denoised image is subtracted from the brightness value of the pixel at the same position in the smoothed image to obtain the edge difference. Finally, the edge difference is multiplied by an enhancement coefficient and added to the brightness value of the pixel at the same position in the denoised image.
[0035] The range and enhancement coefficient of the Gaussian filter are determined based on the parameters and the image. Multiple sets of Gaussian filter ranges and enhancement coefficients are set, and the images used to determine the parameters are de-sharpened for each set. For each set of parameters, the average brightness difference between the two sides of the helmet outline, seat belt outline, protective clothing outline, and personnel outline is calculated, and the proportion of pixels whose brightness reaches the maximum or minimum brightness value of the image after processing is statistically analyzed.
[0036] The average brightness difference on both sides of the contour and the proportion of pixels reaching the maximum or minimum brightness value are normalized separately. The normalized average brightness difference on both sides of the contour is then subtracted from the normalized proportion of pixels reaching the maximum or minimum brightness value. The Gaussian filter range and enhancement coefficients that yield the largest result are selected. The denoised image is then de-sharpened using the selected Gaussian filter range and enhancement coefficients to obtain the edge-enhanced image.
[0037] Specifically, in the desharpening process, the brightness value of a pixel at a certain location in the smoothed image is determined by the weighted average of the brightness values of all pixels within the Gaussian filtering range. Pixels closer to the current pixel have a higher weight, and pixels farther away have a lower weight. The edge difference is the positive or negative difference obtained by subtracting the brightness value of a pixel at the same location in the smoothed image from the current pixel brightness value in the denoised image. The pixel brightness value after desharpening is the sum of the current pixel brightness value in the denoised image, the edge difference, and the enhancement coefficient.
[0038] When calculating the average brightness difference between the two sides of the contour, a pixel band of equal width is set on the side of the contour facing inwards from the marked area and the side facing outwards from the marked area, centered on the manually marked contour. The average brightness within each of the two pixel bands is calculated, and the absolute value of the difference between the two average brightnesses is taken as the brightness difference between the two sides of the corresponding contour. The brightness differences corresponding to the personnel contour, safety helmet contour, safety belt contour, and protective clothing contour are obtained separately, and their average values are calculated as the average brightness difference between the two sides of the contour corresponding to the Gaussian filter range and enhancement coefficient.
[0039] An adaptive histogram equalization process with limited contrast is applied to the edge enhancement image. The edge enhancement image is divided into multiple adjacent rectangular regions, the brightness distribution within each rectangular region is adjusted, and the processing results of adjacent rectangular regions are interpolated.
[0040] The side length of the rectangular region is determined by the dimensions of the helmet marking area in the image. The short side lengths of each helmet marking area are calculated, and the median of all short side lengths is taken as the side length of the rectangular region. When the rectangular region is close to the image boundary and cannot form a complete rectangle, the rectangular region is cropped according to the image boundary.
[0041] The contrast limit value is determined based on the parameters and the image. Multiple contrast limit values are set, and the images used for parameter determination are processed separately. For each contrast limit value, the average brightness difference between the marking areas of the safety helmet, seat belt, and protective clothing and their adjacent background areas is calculated, and the increase in the standard deviation of background pixel brightness before and after processing is also calculated. The average brightness difference and the increase in the standard deviation of background pixel brightness are normalized separately. The normalized average brightness difference is subtracted from the normalized increase in the standard deviation of background pixel brightness, and the contrast limit value corresponding to the largest result is selected. The edge enhancement image is processed using the selected contrast limit value to obtain the contrast-adjusted image.
[0042] Specifically, when performing contrast-limited adaptive histogram equalization on each rectangular region, the number of pixels corresponding to each brightness value within the rectangular region is counted to form a brightness histogram for that region. Pixels exceeding the contrast limit are truncated, and the truncated pixels are evenly distributed among the brightness values in the brightness histogram. Based on the redistributed brightness histogram, a correspondence is established between the original brightness values and the adjusted brightness values, and the brightness of each pixel within the rectangular region is adjusted according to this correspondence.
[0043] When a pixel is located between multiple adjacent rectangular regions, bilinear interpolation is performed on the adjusted brightness values of each rectangular region based on the horizontal and vertical distances between the pixel and the center points of each adjacent rectangular region. The interpolation result is used as the final adjusted brightness value of the pixel, thereby reducing the brightness abrupt changes at the boundaries of adjacent rectangular regions.
[0044] The initial normal lighting image is selected from the images used to determine the parameters. The initial normal lighting image is manually confirmed by on-site personnel based on the actual shooting conditions of the image, and simultaneously meets the following conditions: the outlines of personnel, safety helmets, safety belts, and protective clothing can be determined according to the manual marking results; there are no strong light areas in the image that directly illuminate the camera equipment; the reflections on the equipment surface do not cover the outlines of personnel or safety protective equipment; and there is no dust interference in the image that would prevent the continuous determination of the personnel outline.
[0045] Calculate the average brightness, the percentage of pixels reaching the maximum brightness value, and the percentage of pixels reaching the minimum brightness value for each initial normal lighting image. Obtain the median values of the percentages of pixels reaching the maximum brightness value and the minimum brightness value. Delete initial normal lighting images where the percentage of pixels reaching the maximum brightness value is greater than the corresponding median value and where the outlines of personnel or safety equipment are covered by highlighted areas. Delete initial normal lighting images where the percentage of pixels reaching the minimum brightness value is greater than the corresponding median value and where the outlines of personnel or safety equipment cannot be determined according to the manual marking results. The remaining images after deletion are considered normal lighting images.
[0046] The average brightness of each normal lighting image is calculated, and the median of the average brightness of all normal lighting images is used as the reference brightness. When the number of remaining normal lighting images after deletion is less than the minimum number of normal lighting images, initial normal lighting images are selected again from historical high-risk operation videos generated under the same camera equipment, shooting direction, and zoom state, until the number of normal lighting images reaches the minimum number. The minimum number of normal lighting images is determined by testing multiple candidate numbers, selecting the minimum number of candidate images corresponding to the condition where the change in reference brightness after adding or deleting a normal lighting image does not exceed the stable reference brightness ratio.
[0047] Calculate the current average brightness of each contrast-adjusted image. When the current average brightness is less than the reference brightness, gamma correction is applied to increase the image brightness; when the current average brightness is greater than the reference brightness, gamma correction is applied to decrease the image brightness; when the current average brightness is equal to the reference brightness, no gamma correction is performed.
[0048] Divide the current average brightness and reference brightness by the maximum brightness value of the image to convert them to values between zero and one. Divide the logarithm of the converted reference brightness value by the logarithm of the current average brightness value to obtain the gamma value. When the current average brightness is zero, replace it with the minimum non-zero brightness value of the image; when the current average brightness equals the maximum brightness value of the image, use parameters to determine the gamma value in the image that makes the average brightness of the overly bright image closest to the reference brightness.
[0049] The calculated gamma values are used to exponentiate the brightness of each pixel in the contrast-adjusted image, and the result is restored to the original brightness range of the image. If the pixel brightness after gamma correction is lower than the minimum brightness value, it is set to the minimum brightness value; if it is higher than the maximum brightness value, it is set to the maximum brightness value.
[0050] Specifically, let the value obtained by dividing the brightness of a pixel in the contrast-adjusted image by the maximum brightness value of the image be the current pixel brightness conversion value. Then, the gamma-corrected pixel brightness conversion value is the gamma value of the current pixel brightness conversion value raised to the power of its gamma value. Multiplying the gamma-corrected pixel brightness conversion value by the maximum brightness value of the image yields the pixel brightness restored to the original brightness range of the image.
[0051] For example, if the maximum image brightness is 255, the current average brightness is 64, and the reference brightness is 128, the current average brightness and the reference brightness are each divided by 255, and the gamma value is obtained by dividing the logarithm of the reference brightness conversion value by the logarithm of the current average brightness conversion value. Since the resulting gamma value is less than one, exponentiation of the current pixel brightness conversion value can improve the brightness of low-to-medium brightness pixels.
[0052] The parameters determine the brightness histograms obtained from the normal lighting images in the image set, and the brightness value at the upper quantile position of the brightness histogram in each normal lighting image is obtained. The upper quantile position is determined by testing multiple quantile positions: for each quantile position, the proportion of pixels exceeding the corresponding brightness value in both the normal lighting image and the image with strong light or surface reflection is statistically analyzed, and the quantile position that maximizes the difference in pixel proportions between the two types of images is selected. The median value of the corresponding brightness value at the selected quantile position in each normal lighting image is used as the high-brightness threshold.
[0053] Calculate the proportion of pixels whose brightness exceeds the high brightness threshold in each normally lit image, and use the maximum value of the obtained pixel proportions as the high brightness ratio threshold. Calculate the proportion of pixels whose brightness exceeds the high brightness threshold in the gamma-corrected image. When this pixel proportion is greater than the high brightness ratio threshold, compress the pixels whose brightness exceeds the high brightness threshold; when this pixel proportion is not greater than the high brightness ratio threshold, keep the brightness of each pixel unchanged.
[0054] During brightness compression, the brightness value corresponding to the high brightness threshold remains unchanged. Following the original brightness order from the high brightness threshold to the maximum brightness value, the pixel brightness within this range is linearly mapped to the range between the high brightness threshold and the reference brightness upper limit. The reference brightness upper limit is determined by the median value of the maximum brightness values of all normally illuminated images in the image. If the mapped brightness is higher than the maximum brightness value of the image, it is set to the maximum brightness value of the image.
[0055] Specifically, for pixels whose brightness is not greater than the high brightness threshold, their original brightness remains unchanged. For pixels whose brightness is greater than the high brightness threshold, the ratio of the difference between the pixel's original brightness and the high brightness threshold to the difference between the image's maximum brightness value and the high brightness threshold is first calculated. This ratio is then multiplied by the difference between the reference brightness upper limit and the high brightness threshold, and added to the high brightness threshold to obtain the compressed brightness. This maintains the brightness continuity at the high brightness threshold and preserves the original brightness order of each high-brightness pixel.
[0056] Images that have undergone bilateral filtering, desharpening, adaptive histogram equalization with contrast limiting, gamma correction, and highlight pixel compression are used as preprocessed image frames. These preprocessed image frames are saved in the order of the original frames, and each preprocessed image frame is associated with its corresponding job number, job type, job area boundary, camera equipment number, timestamp, frame number, and the type of safety protective equipment required.
[0057] In step two, the preprocessed image frames generated in step one are read, along with the job number, job type, job area boundary, camera equipment number, timestamp, frame number, and required safety equipment category stored for each preprocessed image frame.
[0058] Before conducting the detection, a YOLOv8 target detection algorithm is established for detecting personnel and safety equipment. The YOLOv8 target detection algorithm extracts features from the input image and outputs the rectangular region containing the personnel and safety equipment, its category, and the detection confidence score. The detection confidence score is a value between zero and one, representing the degree of confidence with which the YOLOv8 target detection algorithm classifies the corresponding rectangular region as a certain category.
[0059] In one embodiment, the YOLOv8 object detection algorithm employs a feature extraction network, a multi-scale feature fusion network, and a detection output network capable of receiving images with three color components. The feature extraction network sequentially extracts image features of different spatial dimensions from the input image; the multi-scale feature fusion network upsamples, downsamples, and concatenates image features of different spatial dimensions, enabling local features of small-sized safety protective equipment to participate in detection along with larger-scale features of the personnel outline; the detection output network outputs the corresponding category, detection box, and detection confidence score for personnel, safety helmets, safety belts, protective clothing, and safety protective equipment, respectively.
[0060] For example, YOLOv8s can be used as the initial network for the YOLOv8 object detection algorithm, with the number of output categories set to the total number of categories such as people, safety helmets, safety belts, protective clothing, and safety equipment, and the network parameters pre-trained on general image data used as initial parameters. When using YOLOv8 object detection algorithms of other scales, the training images, manually labeled categories, and subsequent detection box processing remain unchanged.
[0061] When training the YOLOv8 object detection algorithm, differences in rectangle positions can be determined by variations in the center point position, width, and height of the detection boxes, as well as the degree of overlap. Class differences can be determined by whether the algorithm's output class matches the manually labeled class. Differences in detection confidence can be determined by whether the detection confidence at the corresponding position of the manually labeled rectangle approaches one, and whether the detection confidence in the background region approaches zero. These differences are then used together to adjust the network parameters of the YOLOv8 object detection algorithm.
[0062] In one possible embodiment, training images can be horizontally flipped, have their brightness adjusted, partially occluded, and scaled, but not altered in terms of the actual type or wearing position of the safety equipment. During training, a set of validation images is retained but not used for algorithm parameter adjustments; it is only used to calculate the missed detection rate, false detection rate, and processing time per image.
[0063] Images for training and verification are selected from historical high-risk operation videos. The selected images should originate from the camera equipment actually used to film high-risk operations and include images under normal lighting, low light, strong light, equipment surface reflections, personnel obstruction, and partial obstruction by safety protective equipment. If a specific type of safety protective equipment or a specific filming condition is missing from historical high-risk operation videos, supplementary images containing that safety protective equipment type or filming condition should be collected in the same work area, with the same camera equipment, and under the same filming direction.
[0064] Personnel and safety equipment are manually marked according to their actual positions in the image. Personnel are marked using the smallest rectangle that can enclose the visible part of the human body. Safety helmets, safety belts, protective clothing, and safety protective gear are marked using the smallest rectangle that can enclose the visible part of their respective safety protective equipment. Each rectangle records the x-coordinate of the top left corner, the y-coordinate of the top left corner, the x-coordinate of the bottom right corner, the y-coordinate of the bottom right corner, and the corresponding category. The category name of the safety protective equipment must be consistent with the category of the safety protective equipment required to be worn, as read in step one; different names should not be used for the same safety protective equipment.
[0065] The labeled images are divided into training and validation image groups based on their source videos. The training image group is used to adjust the parameters in the YOLOv8 object detection algorithm, while the validation image group is used to determine the image input size, detection confidence threshold, and overlapping detection box selection criteria. Adjacent image frames from the same high-risk operation video segment are grouped into the same image group to avoid adjacent image frames with essentially the same content appearing in both the training and validation image groups.
[0066] Before inputting the YOLOv8 object detection algorithm, the training and validation images are adjusted according to a pre-selected input image size. During adjustment, the original aspect ratio of the images is maintained. When the adjusted image width or height is smaller than the input image size, image regions with zero pixel values are filled in the insufficient areas to make the processed image width and height equal to the input image size. The image adjustment ratio and the horizontal and vertical fill widths are recorded to restore the detection results to the original coordinates of the pre-processed image frames.
[0067] The image input size was determined using a set of validation images. The YOLOv8 object detection algorithm was trained and validated using multiple candidate image input sizes, and the number of missed detections, false detections, and single-image processing time for personnel and various types of security equipment were statistically analyzed for each candidate image input size. A missed detection refers to a manually labeled person or security equipment in the validation image not generating a corresponding detection box; a false detection refers to a generated detection box that does not correspond to a manually labeled person or security equipment.
[0068] When determining whether a detection bounding box corresponds to a manually marked rectangle, the categories of the detection bounding box and the manually marked rectangle must first be the same. Then, the overlap ratio between the detection bounding box and the manually marked rectangle is calculated. If the overlap ratio is not less than the correct detection matching threshold, the two are considered to correspond. A detection bounding box corresponds to only one manually marked rectangle, and a manually marked rectangle corresponds to only one detection bounding box. When multiple corresponding results exist, the result with the largest overlap ratio is selected first.
[0069] Manually marked rectangles that do not correspond to any detection bounding box are considered missed detections, and detection bounding boxes that do not correspond to any manually marked rectangles are considered false detections. The correct detection matching threshold is determined by the validation image group. Multiple candidate correct detection matching thresholds are set, and the candidate correct detection matching threshold with the smallest sum of the number of missed detections, the number of false detections, and the number of false corresponding detections is selected.
[0070] For each candidate image input size, the number of missed detections is divided by the total number of manually labeled images to obtain the missed detection ratio; the number of false detections is divided by the total number of detection boxes to obtain the false detection ratio. Candidate image input sizes that require processing time longer than the allowed on-site processing time are first excluded. Then, the image input size with the lowest sum of the missed detection ratio and the false detection ratio is selected from the remaining candidate image input sizes. The allowed on-site processing time is determined based on the frame rate of the high-risk operation video and the computing power of the processor used to execute the YOLOv8 object detection algorithm on-site, ensuring that continuously input preprocessed image frames do not accumulate due to insufficient processing speed.
[0071] The parameters of the YOLOv8 object detection algorithm are adjusted using a set of training images. During training, each training image, along with rectangles of people and safety equipment within that image, is input into the YOLOv8 object detection algorithm. The differences between the algorithm's output rectangle positions, categories, and detection confidence scores and the manually labeled results are compared, and the algorithm parameters are repeatedly adjusted based on these differences.
[0072] After each training round, the false negative rate and false positive rate for personnel and various security equipment are calculated using the validation image set. Training stops when the sum of the false negative rate and false positive rate no longer decreases across multiple training rounds, and the algorithm parameters corresponding to the minimum sum of these rates are retained. The number of rounds used to determine the number of consecutive rounds is determined through the training process of the training and validation image sets, selecting the minimum number of rounds that eliminates fluctuations in single-round results and prevents continued training if the validation results remain unchanged for an extended period.
[0073] Specifically, multiple candidate epochs are set. For each candidate epoch, it is checked whether the sum of the missed detection rate and the false detection rate during training that continuously reaches that candidate epoch is not lower than the previous minimum. The number of times training stops prematurely due to errors and the number of epochs that continue training after the validation results remain unchanged for a long time are counted when using each candidate epoch. The candidate epoch with the fewest premature training stops and the fewest continued training epochs is selected.
[0074] For example, the candidate round number can be set to three, five, eight, and ten rounds respectively. When the sum of the missed detection rate and the false detection rate obtained from five consecutive rounds of training is not lower than the previous minimum, and five rounds can eliminate fluctuations in the results of a single round or two adjacent rounds, five rounds are used as the round number to judge multiple consecutive rounds.
[0075] Each preprocessed image frame is adjusted according to the determined image input size and then input into the trained YOLOv8 object detection algorithm. The YOLOv8 object detection algorithm extracts features from the preprocessed image frames and outputs the detection bounding boxes, categories, and detection confidence scores for personnel, safety helmets, safety belts, protective clothing, and safety equipment, respectively. The detection bounding box is a rectangle enclosing the object to be detected, represented by the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner.
[0076] The detection confidence threshold is determined using a set of validation images. Multiple candidate detection confidence thresholds are used to filter the detection results of the validation image set, and the false negative and false positive rates corresponding to each candidate threshold are calculated. The candidate detection confidence threshold with the lowest sum of false negative and false positive rates is selected. Separate detection confidence thresholds are determined for personnel and different categories of safety equipment to avoid applying the same threshold to all categories due to differences in target size and appearance.
[0077] Delete detection boxes whose detection confidence is lower than the corresponding category's detection confidence threshold. For multiple overlapping detection boxes of the same category, calculate the overlap ratio between any two detection boxes. The overlap ratio is the area of the intersection region of the two detection boxes divided by the area occupied by the merged region of the two detection boxes.
[0078] The overlap ratio threshold is determined using a validation image set. Multiple candidate overlap ratio thresholds are applied to process the detection results of the validation image set. When the overlap ratio of two bounding boxes of the same class is greater than the candidate overlap ratio threshold, the bounding box with higher detection confidence is retained, and the bounding box with lower detection confidence is deleted; when the overlap ratio is not greater than the candidate overlap ratio threshold, both bounding boxes are retained. The missed detection ratio and false detection ratio are calculated for each candidate overlap ratio threshold, and the candidate overlap ratio threshold with the smallest sum of the two is selected.
[0079] Based on the image adjustment scale and the horizontal and vertical fill widths, the coordinates of the retained detection boxes are restored to the coordinates of the preprocessed image frame. During coordinate restoration, the horizontal fill width is first subtracted from the horizontal coordinate of the detection box, and the vertical fill width is subtracted from the vertical coordinate of the detection box. The resulting coordinates are then divided by the image adjustment scale. If the restored coordinates are less than the lower limit of the preprocessed image frame coordinates, they are set to the corresponding lower limit; if the restored coordinates are greater than the upper limit of the preprocessed image frame coordinates, they are set to the corresponding upper limit.
[0080] Detection boxes categorized as "personnel" are designated as "personnel detection boxes." A personnel detection box represents a rectangular area in the preprocessed image frame containing a person. Detection boxes categorized as "safety helmets, safety belts, protective clothing, or safety equipment" are designated as "safety equipment detection boxes." A safety equipment detection box represents a rectangular area in the preprocessed image frame containing a piece of safety equipment.
[0081] Extract the color of the work vest from the image area corresponding to each personnel detection frame. The work vest color is determined according to the personnel role identification regulations implemented on site. If the on-site regulations stipulate that red work vests represent supervisors and blue work vests represent work supervisors, and personnel not wearing either of these colors represent workers, then extract the red work vest area and the blue work vest area respectively. If other colors are used on site to distinguish personnel roles, determine the corresponding color and personnel role according to the personnel role identification regulations that have been approved and implemented on site, without changing the meaning of the same color in this high-risk operation.
[0082] The image within the personnel detection frame is converted from three color components (red, green, and blue) to three color components (hue, saturation, and brightness). Hue represents the color type, saturation represents the vividness of the color, and brightness represents the lightness or darkness of the color. These three color components are used together to filter the color of work vests, avoiding the misidentification of gray shadows, strong white light, or low-saturation backgrounds as work vests.
[0083] Specifically, when converting the red, green, and blue color components of each pixel within the personnel detection frame into hue, saturation, and brightness, the maximum and minimum color components among the three color components are first obtained. The maximum color component is used as the basis for calculating brightness, and the ratio of the difference between the maximum and minimum color components to the maximum color component is used as the basis for calculating saturation. The hue is determined based on the differences between the red, green, or blue color components corresponding to the maximum color component and the other color components.
[0084] When the red hue range spans both the start and end positions of the hue value range, the hue ranges near the start and end positions are combined as candidate red hue ranges. A pixel is considered to meet the corresponding working vest condition when its hue is within the defined hue range, its saturation is not less than the defined lower saturation limit, and its brightness is within the defined brightness range.
[0085] Images containing the roles of supervisors, work supervisors, and workers are selected from historical high-risk operation videos captured using the same camera equipment, from the same shooting direction, and under the same zoom setting. The roles of personnel in these images are determined by work permits, on-site personnel arrangement records, or manual verification. Pixels are marked within the work vest areas of both supervisors and work supervisors, and pixels outside the work vests are marked within the surrounding equipment, ground, protective clothing, and background areas.
[0086] Multiple candidate tonal ranges, saturation lower limits, and brightness ranges were set, and the red and blue work vest regions in the image used to determine the character colors were extracted separately. For each set of candidate parameters, the number of correctly extracted work vest pixels and the number of incorrectly extracted non-work vest pixels were counted. The number of correctly extracted work vest pixels was divided by the total number of manually labeled work vest pixels to obtain the work vest pixel extraction ratio; the number of incorrectly extracted non-work vest pixels was divided by the total number of manually labeled non-work vest pixels to obtain the non-work vest pixel mis-extraction ratio. The tonal range, saturation lower limit, and brightness range that resulted in the largest difference between the work vest pixel extraction ratio and the non-work vest pixel mis-extraction ratio were selected for both the red and blue work vests.
[0087] Based on the determined hue range, saturation lower limit, and brightness range, pixels within the personnel detection frame that meet the red work vest condition are set to one, and other pixels are set to zero, forming a red pixel map; pixels that meet the blue work vest condition are set to one, and other pixels are set to zero, forming a blue pixel map.
[0088] The red and blue pixel images are processed separately, first eroded and then dilated. Erosion removes scattered pixels that cannot form continuous regions at their boundaries, while dilation restores the boundaries of the preserved regions after erosion. Erosion and dilation are performed within a rectangular processing area of the same size. This rectangular processing area is determined based on the character's color, using the width of the scattered, incorrectly extracted regions in the image and the minimum width of the actual work vest area. The largest rectangular processing area is selected that can remove the scattered, incorrectly extracted regions without deleting the actual work vest area.
[0089] Specifically, during erosion, the center of the rectangular processing area is sequentially moved to each pixel position in either the red or blue pixel image. When all pixels within the rectangular processing area are set to 1, the pixel at the corresponding center position in the erosion result is set to 1; otherwise, it is set to zero. During dilation, the center of the rectangular processing area is sequentially moved to each pixel position in the erosion result. When at least one pixel within the rectangular processing area is set to 1, the pixel at the corresponding center position in the dilation result is set to 1; otherwise, it is set to zero.
[0090] The rectangular processing area uses an odd number of pixels as its side length, ensuring that the rectangular processing area has a unique center pixel. When the rectangular processing area extends beyond the boundary of the person detection box, pixels outside the person detection box are treated as zero.
[0091] In the processed red and blue pixel images, horizontally, vertically, or diagonally adjacent pixels with the same value are grouped into the same region, and the number of pixels in each region is counted. The region pixel count threshold is determined using the image based on the character's color. The pixel count of the actual work vest region and the scattered erroneously extracted regions are counted separately, and multiple candidate region pixel count thresholds are set. Regions with a pixel count less than the candidate region pixel count threshold are deleted, and the corresponding number of missed deletions in the work vest region and the number of retained scattered erroneously extracted regions are calculated. The candidate region pixel count threshold with the smallest sum of the two is selected.
[0092] Delete areas with fewer pixels than the defined threshold, designate the remaining red areas as red work vest areas, and the remaining blue areas as blue work vest areas. If multiple work vest areas of the same color exist within a single personnel detection frame, add the pixel counts of each work vest area.
[0093] Calculate the proportion of pixels in the red work vest area to the total number of pixels in the personnel detection box, and calculate the proportion of pixels in the blue work vest area to the total number of pixels in the personnel detection box. The work vest area proportion threshold is determined by the image based on the role color. Multiple candidate work vest area proportion thresholds are set, and known personnel roles are judged. The number of incorrect role judgments is counted, and the candidate work vest area proportion threshold with the fewest incorrect role judgments is selected.
[0094] If the proportion of the red work vest area is greater than the work vest area proportion threshold and also greater than the blue work vest area proportion, the personnel role is determined as a supervisor. If the proportion of the blue work vest area is greater than the work vest area proportion threshold and also greater than the red work vest area proportion, the personnel role is determined as a work supervisor. If neither the proportion of the red nor the blue work vest area is greater than the work vest area proportion threshold, the personnel role is determined as a worker. If the proportions of the red and blue work vest areas are equal and both greater than the work vest area proportion threshold, the personnel role is recorded as undetermined, and no personnel role is forcibly assigned in this preprocessed image frame.
[0095] Assign a position between each safety protection equipment detection frame and a personnel detection frame. Calculate the position of the center point of each safety protection equipment detection frame relative to the personnel detection frame. The lateral relative position of the center point of the safety protection equipment detection frame is the lateral distance between the center point and the left boundary of the personnel detection frame divided by the width of the personnel detection frame; the longitudinal relative position is the longitudinal distance between the center point and the top boundary of the personnel detection frame divided by the height of the personnel detection frame.
[0096] The positional range of the safety protection equipment relative to the personnel is determined based on the completed training and verification images. For each type of safety protection equipment, the lateral and longitudinal relative positions of the center point of the detection frame of that type of safety protection equipment within the detection frame of the personnel to which it belongs are statistically analyzed, and the minimum, maximum, minimum, and maximum longitudinal relative positions in the actual marking results are respectively taken as the positional range of that type of safety protection equipment.
[0097] When the center point of a safety protection equipment detection frame is located within a personnel detection frame, and the lateral and longitudinal relative positions of this center point are within the location range corresponding to the safety protection equipment category, the safety protection equipment detection frame is associated with that personnel detection frame. If a safety protection equipment detection frame simultaneously meets the corresponding conditions of multiple personnel detection frames, the ratio of the area of the intersection region between the safety protection equipment detection frame and each personnel detection frame to the area of the safety protection equipment detection frame is calculated, and the frame with the largest ratio is associated with it; if the ratios are the same, the frame with the closest distance to the center point of the safety protection equipment detection frame is associated with it.
[0098] For each preprocessed image frame, the coordinates and detection confidence of the personnel detection box, the personnel role, the coordinates, category and detection confidence of the safety protection equipment detection box, and the correspondence between the safety protection equipment detection box and the personnel detection box are saved, and stored in correspondence with the job number, job type, job area boundary, camera equipment number, timestamp and frame number of the preprocessed image frame.
[0099] It should be noted that during the maintenance of multi-layer scaffolding in power plant boiler furnaces, the maintenance of high-altitude equipment, or the maintenance of confined spaces, workers will continuously move between different working levels, equipment-obstructed areas, and the edges of the camera frame as the maintenance progresses. The direction of temporary lighting, the reflective areas of metal equipment surfaces, the dust conditions on site, and the obstruction relationship between personnel will also constantly change. The safety helmet, safety belt, or protective clothing corresponding to the same person may alternately show phenomena such as decreased detection confidence, reduced detection frame size, equipment position shift, changes in the correspondence between equipment and personnel, and short-term disappearance of detection results in consecutive image frames. Different phenomena may also conflict with each other. For example, the detection confidence of safety protective equipment may decrease, but its positional change may still be consistent with the movement of personnel, or the detection frame size may be small and close to the image boundary, but the detection frame may appear repeatedly in consecutive image frames.
[0100] In this operational scenario, relying solely on any one of the following factors—detection confidence level, detection frame area, personnel overlap, or equipment location range—to make judgments can easily lead to the misidentification of temporary detection anomalies caused by lighting changes, dust interference, short-term obstruction, or image truncation as missing safety protective equipment. It can also incorrectly attribute erroneous detection results caused by equipment reflections, scaffolding components, or adjacent personnel and equipment to the current personnel. When the work order explicitly requires personnel to wear a certain type of safety protective equipment, and the video simultaneously shows changes in target size, personnel overlap, conflicting equipment attributions, and continuous fluctuations in detection status, the same person may be repeatedly judged as wearing, not wearing, or uncertain at adjacent moments. This results in frequent changes in the verification target, generating a large number of invalid verification records, and even overlooking actual instances of not wearing safety protective equipment.
[0101] Therefore, in step three of this embodiment, the coordinates and detection confidence of the personnel detection frame, personnel role, coordinates, category and detection confidence of the safety protection equipment detection frame, the correspondence between the safety protection equipment detection frame and the personnel detection frame, and the job number, job type, job area boundary, camera equipment number, timestamp, and frame number corresponding to the preprocessed image frame are first read. Simultaneously, the corresponding preprocessed image frame and the required safety protection equipment category saved in step one are also read.
[0102] The preprocessed image frames are arranged sequentially according to the camera equipment number, timestamp, and frame sequence number. Multiple consecutive preprocessed image frames are then retrieved backward from the current preprocessed image frame to form a review observation segment. The initial length of the review observation segment is determined based on the frame rate of the high-risk operation video, ensuring that the review observation segment covers the time required for safety protective equipment to move between adjacent positions during normal personnel movement.
[0103] The initial length of the verification observation segment was determined using verification video containing footage of people walking normally, stopping briefly, turning, and interleaving. Multiple candidate initial durations were set, and each candidate initial duration was multiplied by the frame rate of the high-risk operation video and rounded up to obtain the corresponding candidate initial length. Temporary personnel sequences were created using each candidate initial length. The number of times the same person was incorrectly divided into multiple temporary personnel sequences, the number of different people incorrectly connected into the same temporary personnel sequence, and the verification object generation delay were counted. The candidate initial length with the smallest sum of the first two categories and the shortest verification object generation delay was selected.
[0104] For example, if the frame rate of a high-risk operation video is 25 image frames per second, and the determined candidate initial duration is two seconds, the initial length of the review observation segment is 50 preprocessed image frames.
[0105] In the review and observation segment, the person detection boxes in the preceding and following preprocessed image frames are temporarily matched. For each person detection box in the preceding preprocessed image frame, the overlap ratio, center point movement distance, and height change ratio of the person detection box with each person detection box in the following preprocessed image frame are calculated.
[0106] The x-coordinate of the center point of the personnel detection frame is half the sum of the x-coordinates of the left and right boundaries of the frame, and the y-coordinate is half the sum of the y-coordinates of the top and bottom boundaries. The center point movement distance is the straight-line distance between the center points of the two preceding and following personnel detection frames. Dividing the center point movement distance by the diagonal length of the previous personnel detection frame yields the relative movement distance of the personnel. The height change ratio of the personnel detection frame is the absolute value of the difference between the height of the subsequent and previous personnel detection frames divided by the height of the previous frame.
[0107] The temporary correspondence overlap boundary, temporary correspondence movement boundary, and allowable height change range are determined using a verification video with completed identity manual tagging. Multiple sets of candidate temporary correspondence overlap boundaries, candidate temporary correspondence movement boundaries, and candidate allowable height change ranges are set, and temporary correspondence between preceding and following personnel detection boxes is performed according to each parameter combination. The number of times the same person fails to be temporarily corresponded and the number of times different people are incorrectly temporarily corresponded are counted, and the parameter combination with the smallest sum of the two is selected. When multiple parameter combinations have the same sum of corresponding numbers, the parameter combination with a larger temporary correspondence overlap boundary, a smaller temporary correspondence movement boundary, and a smaller allowable height change range is selected.
[0108] If the overlap ratio of the two personnel detection boxes is not less than the temporary corresponding overlap boundary, or the relative movement distance of the personnel is not greater than the temporary corresponding movement boundary, and the height change ratio of the personnel detection boxes is within the allowable range of height change, then the two personnel detection boxes are determined as temporarily corresponding personnel detection boxes.
[0109] When multiple temporary personnel detection boxes exist, first select the personnel detection boxes with the same personnel role; if the personnel roles are the same or one of the personnel roles is uncertain, select the personnel detection box with the largest overlap ratio; if the overlap ratios are the same, select the personnel detection box with the smallest relative movement distance; if the relative movement distances are still the same, select the personnel detection box with the smallest height change ratio.
[0110] Temporarily corresponding personnel detection boxes are connected according to frame number to form a temporary personnel sequence. The temporary personnel sequence is only used to identify the verification objects in the current preprocessed image frame and does not replace the personnel numbers established subsequently.
[0111] For each type of safety protective equipment required in the temporary personnel sequence, obtain the corresponding safety protective equipment detection frame for each personnel detection frame. Record the detection confidence, area, center point position, area ratio with the personnel detection frame, horizontal relative position of the center point, vertical relative position of the center point, and distance from the image boundary of the safety protective equipment detection frame.
[0112] Select a reference image frame from the review observation segment. The personnel detection boxes in the reference image frame must not touch the boundaries of the preprocessed image frame, their overlap with other personnel detection boxes must not exceed the personnel overlap reference value, and the safety protection equipment detection boxes must meet the corresponding conditions of only one personnel detection box. When a corresponding safety protection equipment detection box exists, the center point of that safety protection equipment detection box should also be located within the position range of the corresponding safety protection equipment category determined in step two.
[0113] The baseline value for personnel overlap is determined using the manual verification results of both non-occluded and occluded personnel in the verification video. Multiple candidate values for personnel overlap are set. When the overlap ratio between a personnel detection box and other personnel detection boxes is greater than the candidate value, personnel overlap is determined for that personnel detection box. The number of instances where non-occlusion is mistakenly identified as personnel overlap and the number of instances where occlusion is mistakenly identified as non-occlusion are counted. The candidate value with the smallest sum of these two values is selected as the baseline value for personnel overlap.
[0114] The baseline quantity and predetermined quantile positions are determined using the correct detection results of the same security equipment category in the verification image group. Multiple candidate baseline quantities and multiple sets of candidate quantile positions located on either side of the median value are set. For each set of candidate parameters, correct detection results are selected according to the candidate baseline quantity, and confidence variation intervals, scale variation intervals, and positional activity intervals are formed according to the candidate quantile positions.
[0115] Correct detection results in the verification image group that are not obstructed by personnel, do not touch the boundaries of the preprocessed image frames, and are uniquely assigned to safety protection equipment are considered to be within the corresponding variation range. Incorrect detection results, incorrectly assigned results, and detection results significantly affected by obstruction are considered to be outside the corresponding variation range. The number of correct detection results outside the variation range and the number of abnormal detection results inside the variation range are counted, and the candidate baseline number and candidate quantile position with the smallest sum of these two values are selected.
[0116] When multiple sets of candidate parameters have the same sum of values, the parameter combination with the smaller number of candidate references and the narrower range between the two candidate quantile positions is selected. The sufficient number of reference image frames refers to a number of reference image frames that reaches the determined reference number.
[0117] For example, the number of candidate benchmarks can be set to five, eight, ten, and fifteen, and the candidate quantile positions on both sides of the median can be set to the tenth and ninetieth quantile positions, the fifteenth and eighty-fifth quantile positions, and the twentieth and eightieth quantile positions, respectively.
[0118] The detection results of safety protection equipment in the reference image frame are arranged according to detection confidence level, detection frame area ratio, horizontal relative position of center point, and vertical relative position of center point, respectively. The median value of each arrangement result is obtained, and the values corresponding to the predetermined quantile positions on both sides of the median value are obtained to form the confidence variation range, scale variation range, and positional activity range of the corresponding safety protection equipment category in the current temporary personnel sequence.
[0119] If there are not enough reference image frames in the review observation segment, read the correct detection results of the same safety protection equipment category in the verification image group in step two, and establish the initial confidence change interval, initial scale change interval, and initial position activity interval in the same way. After the number of reference image frames reaches the baseline number, replace the corresponding initial change interval with the change intervals formed by the current temporary personnel sequence.
[0120] For each new preprocessed image frame added, check if it meets the criteria of the baseline image frame. If it does, add the preprocessed image frame to the baseline image frame, and delete the baseline image frame with the earliest timestamp that exceeds the verification observation segment. Then, redetermine the confidence interval, scale interval, and positional activity interval. If it does not meet the criteria, keep the intervals corresponding to the previous preprocessed image frame unchanged to prevent abnormal detection results caused by personnel obstruction, strong light, dust, or image boundary truncation from altering the normal detection baseline.
[0121] Calculate the position of the center point of the safety protection equipment detection frame relative to the center point of the corresponding personnel detection frame, and compare the positional changes in adjacent preprocessed image frames. The lateral movement distance of the safety protection equipment detection frame center point is the x-coordinate of the safety protection equipment detection frame center point in the current preprocessed image frame minus the x-coordinate of the corresponding safety protection equipment detection frame center point in the previous preprocessed image frame. The vertical movement distance of the safety protection equipment detection frame center point is the y-coordinate of the safety protection equipment detection frame center point in the current preprocessed image frame minus the y-coordinate of the corresponding safety protection equipment detection frame center point in the previous preprocessed image frame. Both the lateral and vertical movement distances are retained in their positive and negative signs.
[0122] The horizontal movement distance of the center point of the person detection box is the x-coordinate of the center point of the person detection box in the current preprocessed image frame minus the x-coordinate of the center point of the corresponding person detection box in the previous preprocessed image frame. The vertical movement distance of the center point of the person detection box is the y-coordinate of the center point of the person detection box in the current preprocessed image frame minus the y-coordinate of the center point of the corresponding person detection box in the previous preprocessed image frame.
[0123] The lateral movement difference is obtained by subtracting the lateral movement distance of the corresponding personnel detection frame center point from the lateral movement distance of the safety protection equipment detection frame center point; the longitudinal movement difference is obtained by subtracting the longitudinal movement distance of the corresponding personnel detection frame center point from the longitudinal movement distance of the safety protection equipment detection frame center point. The lateral movement difference is then divided by the width of the previous personnel detection frame, and the longitudinal movement difference is divided by the height of the previous personnel detection frame, allowing for comparison of the lateral and longitudinal movement differences corresponding to different personnel detection frame sizes.
[0124] The median values of the lateral and longitudinal motion differences within the observed segment were obtained separately. The deviation of the current lateral motion difference from the corresponding median value is the absolute value of the difference between the two values, and the deviation of the current longitudinal motion difference from the corresponding median value is the absolute value of the difference between the two values.
[0125] The allowable range of movement is determined using the safety protection equipment detection frames and personnel detection frames that are manually confirmed to belong to the same person in the verification video. Multiple sets of allowable ranges for lateral and longitudinal movement are set separately. The number of safety protection equipment detection frames that move with the personnel but are incorrectly judged as non-moving detection results, and the number of incorrect detection frames that are incorrectly judged as moving with the personnel, are counted. The allowable range for lateral and longitudinal movement with the smallest sum of the two is selected.
[0126] When the deviation of the current lateral follow-up difference from the corresponding median value is within the allowable range of lateral follow-up, and the deviation of the current longitudinal follow-up difference from the corresponding median value is within the allowable range of longitudinal follow-up, the detection result of the safety protection equipment is recorded as the detection result of the movement of personnel.
[0127] A positional shift refers to a change in the lateral or vertical relative position of the center point of the detection frame for safety equipment in adjacent preprocessed image frames that exceeds the positional shift boundary. The positional shift boundary is determined using verification videos. Multiple candidate positional shift boundaries are set, and the number of normal human movements incorrectly identified as positional shifts and the number of incorrect detection results not identified as positional shifts are counted. The candidate positional shift boundary with the smallest sum of these two counts is selected.
[0128] When the position of the safety protection equipment detection frame changes, the safety protection equipment detection frame is recorded as a non-follow-up detection result. If the lateral movement distance of the safety protection equipment detection frame is opposite in sign to the lateral movement distance of the personnel detection frame, or if the longitudinal movement distance of the safety protection equipment detection frame is opposite in sign to the longitudinal movement distance of the personnel detection frame, it is recorded as a movement in the opposite direction. When the number of consecutive frames confirming opposite movement directions is reached, the safety protection equipment detection frame is recorded as a non-follow-up detection result.
[0129] The number of confirmation frames for reversed movement direction is determined using verification video. Multiple candidate confirmation frames for reversed movement direction are set, and the number of normal human movements incorrectly identified as non-follow-up detection results and the number of incorrectly detected results not identified as non-follow-up detection results are counted. The candidate confirmation frame number with the smallest sum of these two is selected. "Successively multiple times" refers to continuously reaching the determined number of confirmation frames for reversed movement direction.
[0130] Within each personnel detection frame, establish the area where the safety protection equipment should appear, according to the location range determined in step two. When the area where the equipment should appear extends beyond the personnel detection frame, crop the portion within the personnel detection frame. Calculate the percentage of overlap between the area where the equipment should appear and other personnel detection frames, and calculate the percentage of the area where the equipment should appear extends beyond the boundaries of the preprocessed image frame.
[0131] Specifically, the location range of the safety protection equipment includes the minimum lateral relative position, the maximum lateral relative position, the minimum longitudinal relative position, and the maximum longitudinal relative position. The minimum and maximum lateral relative positions are multiplied by the width of the personnel detection frame, and then added to the left boundary x-coordinate of the personnel detection frame to obtain the left and right boundary x-coordinates of the area where the equipment should appear. The minimum and maximum longitudinal relative positions are multiplied by the height of the personnel detection frame, and then added to the upper boundary y-coordinate of the personnel detection frame to obtain the upper and lower boundary y-coordinates of the area where the equipment should appear.
[0132] The percentage of the area where the equipment should appear overlaps with other personnel detection frames is the area of the intersection of the equipment should appear area and other personnel detection frames divided by the area of the equipment should appear area. The percentage of the area where the equipment should appear area extends beyond the preprocessed image frame boundary is the area of the equipment should appear area located outside the coordinate range of the preprocessed image frame divided by the area of the equipment should appear area.
[0133] The visible confirmation boundary is determined using manually marked verification images of the area where the equipment should appear, indicating whether it is visible or restricted. Multiple candidate visible confirmation boundaries are set, and the proportion of the effective visible area to the area where the equipment should appear determines whether the area is visible or restricted. The number of times a visible state is incorrectly identified as a restricted state and the number of times a restricted state is incorrectly identified as a visible state are counted, and the candidate visible confirmation boundary with the smallest sum of these two values is selected.
[0134] The portion of the area where the equipment should appear that is not covered by other personnel detection frames and does not exceed the boundary of the preprocessed image frame is recorded as the effective visible area. The ratio of the effective visible area to the area of the area where the equipment should appear is calculated. When the ratio is not less than the visibility confirmation boundary, the current position of the safety protection equipment is recorded as visible; when the ratio is less than the visibility confirmation boundary, the current position of the safety protection equipment is recorded as restricted.
[0135] The average brightness of pixels within the current person detection box, the proportion of pixels whose brightness reaches the high brightness threshold, and the average brightness difference between adjacent preprocessed image frames are calculated. The standard deviation of pixel brightness and the change in the proportion of high-brightness pixels within a predetermined width region surrounding the person detection box are also calculated.
[0136] The predetermined width region around the person detection box is formed by expanding the width of the surrounding region outward from each side of the person detection box and deleting the portion of the expanded region that belongs to the inside of the person detection box. When the expanded region exceeds the boundary of the preprocessed image frame, the portion that is within the preprocessed image frame is truncated.
[0137] The width of the surrounding area is the product of the smaller of the width and height of the personnel detection frame and the ratio of the surrounding area width. The ratio of the surrounding area width is determined using verification video that includes normal lighting variations, strong light, equipment surface reflections, and dust interference. Multiple candidate surrounding area width ratios are set, and the candidate ratio that best covers the changes in the imaging around the personnel and has the smallest area occupied by other personnel detection frames is selected. The predetermined width is the surrounding area width calculated according to the determined surrounding area width ratio.
[0138] The average brightness difference between adjacent preprocessed image frames is the average of the absolute values of the differences between the brightness of each pixel within the current person detection box and the brightness of each pixel within the corresponding person detection box in the previous preprocessed image frame. When the current person detection box and the previous person detection box have different sizes, the image within the previous person detection box is first adjusted to the width and height of the current person detection box before calculating the average brightness difference.
[0139] The change in the proportion of highlighted pixels is the absolute value of the difference between the proportion of highlighted pixels within the person detection box in the current preprocessed image frame and the proportion of highlighted pixels within the corresponding person detection box in the previous preprocessed image frame. The change in the standard deviation of the brightness of pixels in the surrounding area is the absolute value of the difference between the standard deviation of the brightness of pixels in the surrounding area in the current preprocessed image frame and the standard deviation of the brightness of pixels in the corresponding surrounding area in the previous preprocessed image frame.
[0140] The baseline variation range is determined based on the baseline image frame. The average brightness difference, the change in the proportion of bright pixels, and the standard deviation of the brightness of surrounding pixels are calculated for each baseline image frame and arranged in ascending order of value. The value located at a predetermined imaging quantile in each arrangement is obtained, and the range from zero to this value is taken as the baseline variation range for the corresponding indicator.
[0141] The predetermined imaging quantile position is determined using verification video. Multiple candidate imaging quantile positions are set, and the number of frames that are incorrectly judged as imaging state changes when the imaging state is normal, as well as the number of frames that are not judged as imaging state changes due to strong light, equipment surface reflection, or dust interference, are counted. The candidate imaging quantile position with the smallest sum of the two is selected.
[0142] The number of frames required to confirm stable imaging status is determined using verification videos. Among preprocessed image frames that consecutively reach the number of frames required to confirm stable imaging status, if the average brightness difference, the change in the proportion of bright pixels, and the change in the standard deviation of pixel brightness in the surrounding area are all within the corresponding baseline ranges, the corresponding preprocessed image frame is recorded as a frame with stable imaging status. The term "adjacent multiple preprocessed image frames" refers to preprocessed image frames that consecutively reach the determined number of frames required to confirm stable imaging status.
[0143] When the average brightness difference, the change in the proportion of bright pixels, or the change in the standard deviation of pixel brightness in the surrounding area within the current personnel detection frame exceeds the corresponding baseline change range, the current preprocessed image frame is recorded as the imaging state change frame.
[0144] For each safety protection equipment detection frame, check whether its detection confidence is lower than the lower boundary of the current confidence change interval, whether the area ratio of the detection frame is lower than the lower boundary of the current scale change interval, whether the relative position of the center point exceeds the current position activity range, whether it belongs to a non-follow-up detection result, whether it is close to the boundary of the preprocessed image frame, and whether there are multiple corresponding personnel detection frames.
[0145] If the confidence level of the safety protection equipment detection is lower than the lower boundary of the current confidence change interval, but the detection frame of the safety protection equipment is a detection result that moves with the personnel, and the relative position of its center point is within the current location's activity range, the safety protection equipment detection result is not immediately determined as a detection result requiring review. Continue to check whether the decrease in detection confidence occurs synchronously with the change in imaging state frames.
[0146] Synchronous appearance refers to the first time that the detection confidence level falls below the lower boundary of the current confidence change interval, the first time that the detection box area ratio falls below the lower boundary of the current scale change interval, or the first time that the detection box partially disappears. The difference in frame number between the preprocessed image frame and the imaging state change frame is no greater than the number of synchronous confirmation frames.
[0147] The number of synchronization confirmation frames is determined using verification videos. Multiple candidate synchronization confirmation frame numbers are set, and the number of detection anomalies caused by changes in imaging state that are not judged as synchronous occurrences, and the number of detection anomalies unrelated to changes in imaging state that are incorrectly judged as synchronous occurrences are counted. The candidate synchronization confirmation frame number with the smallest sum of the two is selected.
[0148] If the detected anomaly occurs before the frame of the image state change and the difference between the two frame numbers is not greater than the number of synchronous confirmation frames, or if the detected anomaly occurs after the frame of the image state change and the difference between the two frame numbers is not greater than the number of synchronous confirmation frames, it will be treated as a synchronous occurrence.
[0149] If a decrease in detection confidence, a reduction in the proportion of the detection box area, or the partial disappearance of the detection box occurs simultaneously with a frame showing a change in the imaging state, and the frame recovers to the corresponding change range in a subsequent frame showing a stable imaging state, this process is recorded as a temporary imaging fluctuation, and a verification object is not repeatedly established for each preprocessed image frame.
[0150] If the detection confidence decreases or the proportion of the detection frame area decreases without being synchronized with the frame of change in imaging state, or if the detection result of the safety protection equipment still does not recover to the corresponding change range after the imaging state has stabilized, the detection result of the safety protection equipment shall be determined as a continuous abnormal detection result.
[0151] For safety protection equipment inspection frames whose area ratio is lower than the lower boundary of the current scale variation range, check the continuity of the safety protection equipment inspection frame in the review observation segment. If the safety protection equipment inspection frame appears continuously, and it is an inspection result that moves with personnel, and the relative position of its center point remains within the current position's activity range, it is determined to be an inspection result requiring review not only because of its small size.
[0152] If the area of the safety protection equipment detection frame is small, and there are simultaneous changes in position, detection category, correspondence with personnel, or intermittent disappearance in the stable imaging frame, the detection result of the safety protection equipment is determined to be a detection result that needs to be reviewed.
[0153] For safety protection equipment detection frames that do not correspond to personnel detection frames, calculate the intersection area ratio, relative position of the center point, and lateral and longitudinal follow-up differences between the safety protection equipment detection frame and each personnel detection frame. Arrange the personnel detection frames in descending order of intersection area ratio, and obtain the first two personnel detection frames.
[0154] Calculate the difference between the intersection area ratio of the first person detection box and the intersection area ratio of the second person detection box to obtain the intersection area ratio difference value.
[0155] The attribution advantage boundary is determined using the manual verification results of the personnel actually belonging to the safety protection equipment in the verification video. Multiple candidate attribution advantage boundaries are set. When the difference in the intersection area ratio is not less than that of the candidate attribution advantage boundary, the intersection area ratio of the first personnel detection frame is determined to be significantly greater than that of the second personnel detection frame; when the difference in the intersection area ratio is less than that of the candidate attribution advantage boundary, the intersection area ratios of the first two personnel detection frames are determined to be similar.
[0156] The correspondence between safety protection equipment detection frames and personnel detection frames is determined using each candidate attribution advantage boundary. The number of safety protection equipment detection frames that are incorrectly mapped to other personnel detection frames and the number of objects that can be determined but are incorrectly identified as attribution conflict review objects are counted. The candidate attribution advantage boundary with the smallest sum of the two is selected.
[0157] When the intersection area ratio of the personnel detection frame in the first position is significantly larger than that of the personnel detection frame in the second position, and the safety protection equipment detection frame maintains a follow-up relationship with the personnel detection frame in the first position, the safety protection equipment detection frame is reassigned to the personnel detection frame in the first position.
[0158] If the intersection area ratios of the first two personnel detection boxes are similar, or if the safety protection equipment detection box alternately meets the corresponding conditions of the first two personnel detection boxes in consecutive preprocessed image frames, the following relationship between the safety protection equipment detection box and the two personnel detection boxes is compared respectively. If the safety protection equipment detection box maintains a following relationship with only one of the personnel detection boxes, it is re-assigned to that personnel detection box.
[0159] If a safety protection equipment detection frame maintains a follow-up relationship with two personnel detection frames, does not maintain a follow-up relationship with either of the two personnel detection frames, or its follow-up objects repeatedly switch during the review and observation segment, then the safety protection equipment detection frame and the corresponding two personnel detection frames shall be jointly identified as the objects of the conflict of ownership review.
[0160] A follow-up object switching is defined as the object moving from one person detection box to another. When the number of follow-up object switching events within a review observation segment reaches the required number of attribute switching confirmations, repeated switching of the follow-up object is determined. The required number of attribute switching confirmations is determined using verification video. Multiple candidate attribute switching confirmation counts are set, and the number of objects incorrectly identified as having attribute conflicts during normal personnel interaction and the number of objects with actual attribute conflicts that were not determined are counted. The candidate attribute switching confirmation count with the smallest sum of these two counts is selected.
[0161] If a safety protection equipment detection frame is already associated with a personnel detection frame, but its center point is outside the current location's activity range, check whether the detection frame maintains a following relationship with other personnel detection frames. If it does, cancel the original association and recheck the relative position of its center point according to the other personnel detection frames. If, after rechecking, it is within the corresponding location's activity range, reassign the safety protection equipment detection frame to the other personnel detection frames; if it is still not within the location's activity range, the safety protection equipment detection result is identified as a location anomaly for review.
[0162] When the detection frame of the safety protection equipment approaches the boundary of the preprocessed image frame, the direction in which the detection frame approaches the boundary and the direction in which the area where the equipment should appear exceeds the boundary of the preprocessed image frame are checked respectively. If both directions are the same, and the detection frame of the safety protection equipment moves continuously along the boundary direction in adjacent preprocessed image frames, the detection result of the safety protection equipment is recorded as a boundary continuation state.
[0163] The distance between the safety protection equipment detection frame and the left boundary of the preprocessed image frame is the x-coordinate of the left boundary of the safety protection equipment detection frame; the distance between the safety protection equipment detection frame and the right boundary of the preprocessed image frame is the image width minus the x-coordinate of the right boundary of the safety protection equipment detection frame; the distance between the safety protection equipment detection frame and the top boundary of the preprocessed image frame is the y-coordinate of the top boundary of the safety protection equipment detection frame; and the distance between the safety protection equipment detection frame and the bottom boundary of the preprocessed image frame is the image height minus the y-coordinate of the bottom boundary of the safety protection equipment detection frame.
[0164] When any of the above distances is not greater than the boundary proximity distance, the safety protection equipment detection box is determined to be close to the boundary of the preprocessed image frame corresponding to that distance. The boundary proximity distance is the product of the smaller value of the personnel detection box width and the personnel detection box height and the boundary proximity ratio. The boundary proximity ratio is determined using the process of the safety protection equipment normally entering and leaving the preprocessed image frame in the verification video. Candidate boundary proximity ratios are selected that can identify the boundary continuation state before the safety protection equipment detection box disappears and will not judge detection boxes far from the image boundary as close to the preprocessed image frame boundary.
[0165] If the safety protection equipment detection frame, which is in a boundary continuation state, disappears in subsequent preprocessed image frames, and the corresponding equipment should still partially extend beyond the same image boundary, its disappearance is not directly determined as a failure to detect safety protection equipment. If the safety protection equipment detection frame still does not appear after the equipment should re-enter the preprocessed image frame completely, the corresponding personnel detection frame is determined as a missing verification object after boundary restoration.
[0166] When a certain type of safety protective equipment is required, but the current personnel detection frame does not correspond to the detection frame of that type of safety protective equipment, check whether the area where the equipment should appear is visible, and check the appearance process of the detection frame of that type of safety protective equipment in the review and observation segment.
[0167] When the area where the equipment should be located is in a restricted state, the personnel detection frame is marked as the object to be observed further, and the review observation segment continues to be expanded forward. The number of image frames expanded each time is determined by the number of image frames experienced during the most recent recovery from a restricted state to a visible state. If no record of recovery from a restricted state to a visible state has yet appeared, the expansion is carried out at half the length of the initial review observation segment.
[0168] After the observation segment is expanded, if the area where the equipment should appear becomes visible again and the corresponding safety protection equipment is detected, the personnel detection frame is removed from the pending observation status. If the area where the equipment should appear remains restricted after expanding to the maximum observation length, the personnel detection frame is identified as a verification object with insufficient visibility.
[0169] The maximum observation length is determined by the number of image frames required for the area where the equipment should appear to recover from a restricted state to a visible state in the verification video. The number of image frames required for each recovery from a restricted state to a visible state in the verification video is counted, and the images are arranged in ascending order of the number of image frames. The number of image frames corresponding to the position with the maximum observation quantile is taken as the maximum observation length.
[0170] The maximum observation quantile position is determined by testing multiple candidate quantile positions separately. For each candidate quantile position, the number of images that did not recover to a visible state before reaching the corresponding maximum observation length but subsequently recovered to a visible state, and the number of image frames whose generation of the verification object was delayed due to the excessively long maximum observation length are counted. The candidate quantile position with the smallest sum of the two is selected.
[0171] When the verification video does not show a process of recovering from a restricted state to a visible state, a predetermined multiple of the initial length of the verification observation segment is used as the initial maximum observation length. This predetermined multiple is determined by dividing the on-site allowable verification delay time by the initial duration of the verification observation segment, and then rounding up.
[0172] When the area where the equipment should appear is visible, and the current preprocessed image frame is a stable imaging frame, count the number of consecutive image frames in the review observation segment where the corresponding safety protection equipment is not detected. When the number of consecutive undetected image frames reaches the current missing confirmation length, the detection result of the safety protection equipment is recorded as a stable missing state, and the corresponding personnel detection frame is identified as the missing review object.
[0173] The current missing confirmation length is determined based on the state switching interval of the corresponding safety protection equipment detection results within the review and observation segment. The median value of the number of image frames taken for the corresponding safety protection equipment to change from detected to undetected, and from undetected back to detected, is taken as the temporary disappearance length. The smallest integer number of frames greater than the temporary disappearance length is taken as the current missing confirmation length.
[0174] If no state transition process suitable for determining the temporary disappearance length appears within the reviewed observation segment, the median of the number of disappearance frames of the same safety protection equipment category under short-term occlusion conditions in the verification video is used to determine the initial missing confirmation length. After obtaining the state transition process of the current temporary personnel sequence, the current missing confirmation length is used to replace the initial missing confirmation length.
[0175] For safety equipment required by the work permit but not currently detected, if its disappearance occurs simultaneously with a change in imaging status, and the area where the equipment should appear remains visible, the review observation segment is extended until a stable imaging frame appears. If the corresponding safety equipment is detected again after the imaging status stabilizes, the aforementioned disappearance process is recorded as a temporary absence caused by imaging fluctuations; if it is still not detected, the corresponding personnel detection frame is identified as the object for stable status absence review.
[0176] For newly appearing safety protection equipment detection frames within the personnel detection frame, check whether their first appearance time is synchronized with the expansion of the highlighted pixel area within the personnel detection frame, and check whether the safety protection equipment detection frame is a detection result that moves with the personnel.
[0177] Within the personnel detection frame, pixels with brightness exceeding the high-brightness threshold are set to one, while other pixels are set to zero, forming a high-brightness pixel map. In this high-brightness pixel map, pixels with values of one that are horizontally, vertically, or diagonally adjacent are grouped into the same high-brightness pixel region, and the area and center point position of each high-brightness pixel region are calculated.
[0178] The highlighted pixel regions in the current preprocessed image frame are matched with the highlighted pixel regions in the previous preprocessed image frame according to their overlap ratio. Among the corresponding highlighted pixel regions, if the difference between the area of the current highlighted pixel region and the area of the previous highlighted pixel region is greater than the boundary of the highlighted region area change, the highlighted pixel region is determined to have expanded; if the difference is greater than the boundary of the highlighted region area change, the highlighted pixel region is determined to have shrunk.
[0179] The boundary of the area change of the bright area is determined using verification videos with and without device surface reflection. Candidate boundary of the area change of the bright area are selected that can identify the expansion and contraction of the reflective area and will not judge small-scale bright changes caused by normal human movement as the expansion or contraction of the bright pixel area.
[0180] The movement of the center point of the safety protection equipment detection frame along with the highlighted area means that the distance between the center point of the safety protection equipment detection frame and the center point of the corresponding highlighted pixel area is within the highlighted follow-up distance range in consecutive pre-processed image frames, and the movement direction of the safety protection equipment detection frame and the highlighted pixel area is the same. The highlighted follow-up distance range is determined by verification video showing reflections on the equipment surface.
[0181] The safety protection equipment detection frame only appears during the expansion of the bright pixel area, its center point moves with the bright area but not with the personnel detection frame, and disappears after the bright pixel area shrinks. This safety protection equipment detection frame is then identified as a candidate for reflective interference. For multiple detection frames formed by the same reflective interference candidate in consecutive preprocessed image frames, only the preprocessed image frame with the highest detection confidence is retained as the detection result requiring review, avoiding repeated review of the same reflective area.
[0182] If the detection confidence of the safety protection equipment detection frame decreases during a period of dust state change, but the direction of change in its follow-up relationship, the relative position of the center point, and the proportion of the detection frame area is consistent with the direction of change of similar safety protection equipment among adjacent personnel, the corresponding detection result will be recorded as a common imaging disturbance. Detection results that do not recover after the common imaging disturbance ends are determined to be detection results that require review; detection results that recover after the common imaging disturbance ends will not be reviewed again.
[0183] The changes in dust conditions are determined based on the combined changes in the areas surrounding multiple personnel detection boxes in the preprocessed image frame and the work area image. The average brightness difference, the change in pixel brightness standard deviation, and the change in the number of edge pixels in the work area image are calculated separately. The number of personnel in the same preprocessed image frame whose pixel brightness standard deviation changes in the areas surrounding personnel detection boxes exceed the corresponding baseline change range is also counted.
[0184] When the average brightness difference, pixel brightness standard deviation change, or edge pixel number change in the image of the work area exceeds the dust change range, and the pixel brightness standard deviation changes in the area surrounding the personnel detection box that affects the number of personnel in the same preprocessed image frame, the corresponding preprocessed image frame is recorded as a dust state change frame. Continuous dust state change frames form a dust state change period.
[0185] The range of dust variation and the number of personnel affected by dust were determined using verification videos that included both normal imaging states and dust interference states. The parameter combination with the smallest sum of the number of people whose normal imaging states were incorrectly identified as dust state changes and the number of people whose dust interference states were not identified as dust state changes was selected.
[0186] The term "adjacent personnel" refers to personnel whose distance from the center point of the personnel detection frame is no greater than the adjacent personnel distance boundary. The adjacent personnel distance boundary is determined using verification video, selecting candidate adjacent personnel distance boundaries that include personnel affected by the same dust area and excluding those that are too far away.
[0187] The consistent direction of change for similar safety protective equipment among adjacent personnel means that, among the test results of at least a number of similar safety protective equipment with common changes, the positive and negative directions of the change in confidence level, the positive and negative directions of the change in the relative position of the center point, or the positive and negative directions of the change in the proportion of the test frame area are the same. The number of common changes is determined using a dust interference verification video.
[0188] The termination of common imaging disturbance means that the preprocessed image frames that have continuously reached the number of confirmed stable imaging states no longer belong to the dust state change frames, and the standard deviation of pixel brightness change in the area around the corresponding personnel detection box has recovered to the corresponding baseline change range.
[0189] Within the review and observation segment, the states experienced by each safety protection equipment test result are recorded. These states include normal correspondence, temporary imaging fluctuation, continuous abnormality, boundary continuation, attribution conflict, location abnormality, pending continued observation, insufficient visibility conditions, lack of stable state, and candidate for reflective interference.
[0190] Normal correspondence means that the safety protection equipment detection frame corresponds to a unique personnel detection frame, the detection confidence level is within the current confidence change range, the area ratio of the detection frame is within the current scale change range, the relative position of the center point is within the current position activity range, the safety protection equipment detection frame is a detection result that moves with the personnel, and there are no boundary continuations, ownership conflicts, abnormal positions, candidates for continued observation, or reflective interference.
[0191] Temporary imaging fluctuations refer to the simultaneous occurrence of decreased detection confidence, reduced detection frame area ratio, or partial disappearance of the detection frame with changes in imaging state, and the return of the safety protection equipment detection results to the corresponding change range after the imaging state stabilizes.
[0192] A persistent anomaly refers to a situation where the confidence level of the safety protection equipment is consistently lower than the lower boundary of the current confidence change interval or the area ratio of the detection frame is consistently lower than the lower boundary of the current scale change interval, and the detection result does not appear synchronously with the change in imaging state, or does not return to the corresponding change interval after the imaging state has stabilized.
[0193] Boundary continuation refers to the situation where the detection frame of the security protection equipment approaches the boundary of the preprocessed image frame, and the area of the equipment should extend beyond the boundary of the preprocessed image frame in the same direction, and the detection frame of the security protection equipment should move continuously along that direction.
[0194] A conflict of ownership refers to a situation where the same safety protection equipment detection frame simultaneously meets the corresponding conditions of at least two personnel detection frames, and the personnel detection frame to which it belongs cannot be uniquely determined based on the ratio of the intersecting areas and the following relationship.
[0195] An abnormal position refers to a situation where a safety protection equipment detection frame has been matched with a personnel detection frame, but the relative position of the center point is outside the current position's activity range, and it cannot be re-matched to another personnel detection frame that places the relative position of the center point within the corresponding position's activity range.
[0196] "Pending continued observation" refers to situations where a certain type of safety protective equipment is required, but the current personnel detection frame does not correspond to the detection frame of that type of safety protective equipment, and the area where that type of safety protective equipment should appear is in a restricted state, but the review observation segment has not yet reached the maximum observation length.
[0197] It is evident that insufficient conditions refer to the situation where, after the verification observation segment reaches the maximum observation length, the area where the corresponding safety protection equipment should appear remains restricted.
[0198] A lack of stable state means that the area where the corresponding safety protection equipment should be visible is in a visible state, the imaging state has returned to stability, and the number of consecutive image frames that have not detected the corresponding safety protection equipment has reached the current missing confirmation length.
[0199] Reflective interference candidates refer to the safety protection equipment detection frame that only appears during the expansion of the bright pixel area, moves with the bright pixel area but not with the personnel detection frame, and disappears after the bright pixel area shrinks.
[0200] When the test results of safety protection equipment change directly from a normal state to a state of continuous abnormality, attribution conflict, or location abnormality, it should be immediately identified as a test result requiring review. When the test results of safety protection equipment enter a state of temporary imaging fluctuation, boundary continuation, or pending continued observation, review should not be carried out immediately, and a decision on whether to generate a review object should be made based on subsequent state changes.
[0201] When a temporary imaging fluctuation returns to the normal state, the verification mark of the detection result is cancelled; when a temporary imaging fluctuation turns into a continuous abnormal state, the verification object will be established starting from the preprocessed image frame where the detection confidence level decreases, the detection box area ratio decreases, or the detection box disappears for the first time.
[0202] When the boundary continuation state is restored to the normal corresponding state, no verification object is established; when the boundary continuation state changes to the missing state after boundary restoration, the first image frame in which the equipment should appear in the preprocessed image frame is determined as the verification starting image frame.
[0203] When the pending observation status returns to the normal corresponding status, the pending observation is cancelled; when the pending observation changes to insufficient visibility conditions or missing stable status, the corresponding personnel detection box is determined as the personnel detection box that needs to be reviewed.
[0204] Based on the resulting state, verification objects are generated separately. Single equipment verification objects are generated from continuous anomaly detection results and location anomaly detection results; grouped verification objects containing one safety protection equipment detection frame and at least two personnel detection frames are generated from attribution conflict detection results; missing verification objects centered on personnel detection frames and the areas where equipment should appear are generated from stable state missing and boundary recovery missing; interference elimination verification objects containing safety protection equipment detection frames and corresponding highlighted areas are generated from reflective interference candidate verification objects; and continuous image verification objects containing personnel detection frames and verification observation segments are generated from insufficient visibility conditions.
[0205] For continuous image verification targets, preprocessed image frames within the verification observation segment are read sequentially according to timestamp and frame number. For each preprocessed image frame, a verification image region is formed based on the personnel detection bounding box in that preprocessed image frame, and then sequentially input into the Faster R-CNN object detection algorithm. When the corresponding safety protection equipment is detected in a preprocessed image frame where the area where any equipment should appear within the verification observation segment is visible, the verification detection result and its timestamp are saved; when no corresponding safety protection equipment is detected in any preprocessed image frames where the area where any equipment should appear within the verification observation segment is visible, the continuous verification result of no detection of the corresponding safety protection equipment is saved.
[0206] For groups of objects to be reviewed, a review image area is formed that simultaneously includes the safety protection equipment detection frame and the detection frames of all personnel involved. The left boundary of the review image area is taken as the minimum value of the left horizontal coordinate of each detection frame, the right boundary is taken as the maximum value of the right horizontal coordinate of each detection frame, the upper boundary is taken as the minimum value of the upper vertical coordinate of each detection frame, and the lower boundary is taken as the maximum value of the lower vertical coordinate of each detection frame. Then, it is expanded outward according to the horizontal expansion ratio and the vertical expansion ratio.
[0207] The accompanying state is used to determine the comparison method between the verification result and the original detection result. When the primary verification state is a conflict of ownership, the intersection area ratio, relative position of the center point, and follow-up relationship between the verification detection box and the detection boxes of each involved person are calculated separately to redetermine the correspondence. When the primary verification state is a missing stable state or a missing state after boundary recovery, it is checked whether the Faster R-CNN object detection algorithm outputs a detection box for safety protective equipment required by the work order but not present in the original detection result. When the primary verification state is a persistent anomaly or a location anomaly, the verification detection box is compared with the original detection box in terms of category, overlap ratio, and detection confidence. When the primary verification state is a candidate for reflective interference, if the Faster R-CNN object detection algorithm does not output a verification detection box that matches the corresponding safety protective equipment category and location range, the original detection box is deleted.
[0208] When a personnel detection frame corresponds to multiple verification states simultaneously, the verification states are not simply merged. Instead, the verification state corresponding to each type of safety protection equipment is saved separately, along with the timestamp of the first occurrence of the state, the number of image frames for which the state lasts, the area where the equipment should appear, the follow-up relationship, the confidence change interval, the scale change interval, the positional activity interval, the imaging state, and any other personnel detection frames involved.
[0209] When the same safety protection equipment detection frame simultaneously meets multiple verification states, the primary verification state is determined in the following order: ownership conflict, missing stable state, continuous anomaly, positional anomaly, missing after boundary recovery, candidate for reflective interference, and insufficient visibility conditions. The remaining states are saved as accompanying states. The primary verification state is used to determine the subsequent verification image area, and the accompanying states are used to determine the comparison method between the verification result and the original detection result.
[0210] Save the personnel detection frames that need to be reviewed, safety protection equipment detection frames, areas where equipment should appear, review status, temporary personnel sequence, start and end times of review observation segments, adjacent personnel detection frames involved, and records of changes in each detection result in the review observation segment, and store them in correspondence with the corresponding job number, job type, camera equipment number, timestamp, and frame sequence number.
[0211] A Faster R-CNN object detection algorithm is established. The Faster R-CNN object detection algorithm first generates candidate rectangular regions in the input image that may contain people or safety equipment, and then performs category determination and bounding box position adjustment on each candidate rectangular region. The Faster R-CNN object detection algorithm outputs the bounding boxes, categories, and detection confidence scores for people, helmets, seat belts, protective clothing, and safety equipment.
[0212] In one possible embodiment, the Faster R-CNN object detection algorithm employs a feature extraction network, a candidate region generation network, a region of interest (ROI) pooling network, and a class determination and bounding box position adjustment network. The feature extraction network can use a ResNet50 network and can use network parameters pre-trained on general image data as initial parameters. When using other feature extraction networks, the generation of candidate rectangular regions, ROI pooling, and the verification of bounding box output remain unchanged.
[0213] The input size of the Faster R-CNN object detection algorithm is determined using a set of validation images. Multiple candidate input sizes are set, and the missed detection rate, false detection rate, single-image processing time for the verification image region, and memory usage are statistically analyzed for each candidate input size. Candidate input sizes that have a single-image processing time exceeding the allowable on-site verification processing time or have memory usage exceeding the available memory of the on-site processor are first eliminated. Then, the candidate input size with the lowest sum of the missed detection rate and false detection rate is selected.
[0214] The Faster R-CNN object detection algorithm was trained and validated using the training and validation image sets from Step 2. To increase training samples for small-sized and occluded conditions, images of safety equipment, people occluding each other, partially occluded safety equipment, and safety equipment near image boundaries (with detection box areas no larger than the small-sized threshold) were extracted from the training image set. The extracted images still used the manual labeling results from Step 2, and the category names of people and safety equipment were not changed.
[0215] The area ratio of the safety protection equipment detection frame is calculated by dividing the area of the safety protection equipment detection frame by the area of the corresponding personnel detection frame. The small-size judgment threshold is determined using a verification image group. The correct marking results of the safety protection equipment in the verification image group are arranged in ascending order of the safety protection equipment detection frame area ratio, and multiple candidate small-size judgment thresholds are set for each group.
[0216] For each candidate small size judgment threshold, the number of missed detections when using the YOLOv8 object detection algorithm with a statistical area ratio not exceeding that candidate small size judgment threshold, and the reduction in the number of missed detections after adding the corresponding image to the Faster R-CNN object detection algorithm for training. A candidate small size judgment threshold is selected that can cover small-sized security equipment appearing in the YOLOv8 object detection algorithm's missed detection set, without causing the number of extracted images to exceed a predetermined proportion of the training image group.
[0217] After the Faster R-CNN object detection algorithm extracts features from the image, a candidate region generation network generates candidate rectangular regions on the extracted image features. The candidate region generation network is a convolutional neural network that sets reference rectangles of various sizes and aspect ratios at different locations of image features and determines whether personnel or safety equipment may exist in each reference rectangle.
[0218] The dimensions and aspect ratio of the reference rectangles are determined based on the personnel and safety equipment bounding boxes in the training image set. The width, height, area, and width-to-height ratio of each bounding box are calculated, and the boxes are categorized according to personnel, safety helmets, safety belts, protective clothing, and safety equipment. For each category, multiple area values that can cover small, medium, and large bounding boxes are selected from the distribution of bounding box areas in that category, arranged from smallest to largest. The width and height of the reference rectangles are then determined based on the width-to-height ratio of the corresponding bounding boxes. The selected reference rectangles should ensure that each bounding box in the training image set can overlap with at least one reference rectangle.
[0219] Specifically, the area of the marking frames for personnel, safety helmets, safety belts, protective clothing, and safety equipment, as well as the width-to-height ratio, are grouped separately. For each category, at least one reference rectangle size is selected that approximates the smaller, middle, and larger portions of the marking frame area for that category. The width and height of the reference rectangles are calculated based on the selected area values and the corresponding width-to-height ratios.
[0220] To determine whether a bounding box is covered by a reference rectangle, the center of the reference rectangle is moved to the center of the bounding box, and the overlap ratio between the reference rectangle and the bounding box is calculated. If the overlap ratio is not less than the reference rectangle coverage threshold, the bounding box is considered to form a valid overlapping area with the reference rectangle. The reference rectangle coverage threshold is determined through a set of validation images, selecting the candidate reference rectangle coverage threshold that minimizes the number of manually marked targets not covered by candidate rectangle regions and also minimizes the total number of candidate rectangle regions.
[0221] The candidate region generation network outputs the confidence level of each candidate rectangular region containing the target and the position adjustment amount of the candidate rectangular region. Candidate rectangular regions with a confidence level below the candidate region retention threshold are deleted. The candidate region retention threshold is determined using a set of validation images. Multiple candidate values are processed on the validation images, and the number of manually labeled targets not covered by any candidate rectangular regions and the total number of retained candidate rectangular regions are counted. Candidate values where manually labeled targets are not covered are first excluded; then, the candidate value with the minimum total number of retained candidate rectangular regions is selected as the candidate region retention threshold.
[0222] The retained candidate rectangular regions are subjected to Region of Interest (ROI) pooling. ROI pooling divides candidate rectangular regions of different widths and heights into an equal number of smaller regions corresponding to their image features. A value representing the feature of each smaller region is then extracted, transforming candidate rectangular regions of different sizes into features of the same size. The features processed by ROI pooling are used to determine the category of the candidate rectangular regions and adjust their boundary positions.
[0223] The parameters of the Faster R-CNN object detection algorithm were repeatedly adjusted using the training image set. After each training round, the false negative rate and false positive rate for personnel and various security equipment were calculated using the validation image set. Training was stopped when the sum of the false negative rate and false positive rate no longer decreased after multiple training rounds, and the algorithm parameters corresponding to the minimum sum of the ratios were retained. The number of consecutive rounds was determined using the same method as used in step two for training the YOLOv8 object detection algorithm.
[0224] For personnel detection boxes that require verification, each side of the personnel detection box is expanded outwards by the verification boundary width to obtain the verification image area. The horizontal verification boundary width is the product of the personnel detection box width and the horizontal expansion ratio, and the vertical verification boundary width is the product of the personnel detection box height and the vertical expansion ratio. When the expanded boundary exceeds the preprocessed image frame, the excess portion is truncated to the preprocessed image frame boundary.
[0225] The horizontal and vertical expansion ratios are determined using a set of verification images. Multiple candidate horizontal and vertical expansion ratios are set, and it is determined whether the expanded verification image area completely includes the manually marked safety equipment corresponding to the person. The area occupied by other people in the verification image area is also calculated. First, parameter combinations that cannot completely include the corresponding safety equipment are eliminated. Then, the parameter combination with the smallest area occupied by other people is selected as the horizontal and vertical expansion ratios.
[0226] For safety protection equipment detection frames that do not correspond to personnel detection frames, each side of the safety protection equipment detection frame is expanded outward by a predetermined proportion of the frame's width and height to form a verification image area. The predetermined proportions are determined using a set of verification images, with multiple candidate proportions set, and the candidate proportion that completely includes the manually marked safety protection equipment and has the smallest verification image area is selected.
[0227] Maintaining the aspect ratio of the verification image region, adjust it to the input size of the Faster R-CNN object detection algorithm, and record the adjustment ratio and the horizontal and vertical padding widths. Input the adjusted verification image region into the trained Faster R-CNN object detection algorithm to obtain the verification bounding box, verification category, and verification detection confidence. The verification bounding box is the detection box for personnel or safety equipment output by the Faster R-CNN object detection algorithm in the verification image region.
[0228] Determine the verification detection confidence thresholds for personnel and various security equipment output by the Faster R-CNN object detection algorithm. The verification detection confidence thresholds are determined using a set of validation images, in the same way as the YOLOv8 object detection algorithm in step two. That is, multiple candidate verification detection confidence thresholds are tested, and the candidate verification detection confidence threshold with the smallest sum of the missed detection ratio and the false detection ratio is selected.
[0229] Remove check boxes whose check detection confidence is lower than the corresponding category check detection confidence threshold. Based on the position of the check image region in the preprocessed image frame, the image adjustment ratio, and the horizontal and vertical fill widths, restore the coordinates of the retained check boxes to the coordinates of the preprocessed image frame.
[0230] The check box is compared with the original check box that needs to be checked. If the check box and the original check box are of the same category, and their overlap ratio is greater than the check matching threshold, the coordinates, category, and check confidence score of the check box are used to replace the coordinates, category, and check confidence score of the original check box. If the check box and the original check box are of different categories, and the check confidence score is greater than or equal to the check confidence score threshold for the corresponding category, the category, coordinates, and check confidence score of the check box are used to replace the original check result.
[0231] The verification matching threshold is determined using a set of verification images. Multiple candidate verification matching thresholds are set, and the number of times the same target is incorrectly identified as different targets, and the number of times different targets are incorrectly identified as the same target, are counted. The candidate verification matching threshold with the smallest sum of these two values is selected.
[0232] When the Faster R-CNN object detection algorithm fails to output a corresponding check box of the same class in the check image region whose overlap with the original check box exceeds the check matching threshold, it checks the detection confidence of the original check box. If the detection confidence of the original check box is greater than or equal to the check retention threshold for the corresponding class, the original check box is retained; if the detection confidence of the original check box is less than the check retention threshold, the original check box is deleted. The check retention threshold is determined using a set of validation images. Multiple candidate check retention thresholds are set, and the candidate check retention threshold with the smallest sum of incorrect retentions and incorrect deletions is selected.
[0233] For personnel detection boxes that require the wearing of a certain type of safety protective equipment but were not detected in step two, check whether the Faster R-CNN object detection algorithm outputs a detection box for that type of safety protective equipment within the corresponding verification image region of the personnel. If there is a detection box with a verification detection confidence score greater than or equal to the corresponding category verification detection confidence score threshold, and the center point of the detection box is located within the position range of the corresponding safety protective equipment category determined in step two, then the detection box is associated with the personnel detection box.
[0234] Arrange consecutive preprocessed image frames generated by the same camera device according to timestamp and frame number, and assign the same person to each adjacent preprocessed image frame. For each person detection box in the previous preprocessed image frame, calculate the overlap ratio between it and the person detection boxes in the subsequent preprocessed image frame, and calculate the distance between the center points of the two person detection boxes. Divide the distance between the center points by the diagonal length of the previous person detection box to obtain the relative distance between the center points. The diagonal length of the person detection box is calculated based on the width and height of the person detection box.
[0235] The personnel matching threshold and the center point relative distance threshold are determined using verification videos containing continuous personnel activity. Multiple sets of candidate personnel matching thresholds and candidate center point relative distance thresholds are set separately. Two personnel detection boxes are considered as matchable if the overlap ratio of two consecutive personnel detection boxes is greater than or equal to the candidate personnel matching threshold, or if the center point relative distance is less than or equal to the candidate center point relative distance threshold. The number of incorrectly matched different personnel and the number of unmatched individuals in the verification video are counted, and the threshold combination with the smallest sum of these two counts is selected.
[0236] When multiple corresponding person detection boxes exist for the same person in adjacent preprocessed image frames, the person detection box with the largest overlap ratio is selected first; if the overlap ratios are the same, the person detection box with the smallest relative distance between its center points is selected. Each person detection box can only correspond to one person detection box in the previous preprocessed image frame. The first person detection box that does not correspond to a person detection box in the previous preprocessed image frame is assigned a new person number; corresponding person detection boxes use the same person number.
[0237] If a person is not detected in any of the preprocessed image frames (not exceeding the allowed number of consecutive missing frames) and subsequently reappears, the relative distance between the center points of the reappearing person detection box and the last person detection box for that person is calculated, and it is checked whether the person roles corresponding to the two detection boxes are the same. If the relative distance between the center points is not greater than the re-corresponding distance threshold and the person roles are the same, or if one of the person roles is uncertain, the original person number is used.
[0238] The re-correspondence distance threshold and the allowed number of consecutively missing image frames are determined using verification videos. Multiple candidate re-correspondence distance thresholds and candidate image frame numbers are tested separately. The number of times the same person is incorrectly assigned a different person ID and the number of times different people are incorrectly assigned the same person ID are counted. The parameter combination with the smallest sum of these two counts is selected. When the number of consecutively missing image frames exceeds the determined limit, the consecutive recording corresponding to that person ID is terminated.
[0239] For each personnel number, arrange the personnel detection frames and safety equipment detection frames corresponding to that personnel according to the timestamp. For each type of safety equipment required to be worn as determined in step one, determine the status of the safety equipment in each preprocessed image frame. The safety equipment status includes detected, not detected, and cannot be determined.
[0240] In a preprocessed image frame, if a safety protection equipment detection frame exists corresponding to the personnel detection frame, and the center point of the safety protection equipment detection frame is within the location range of the corresponding safety protection equipment category, the status of that type of safety protection equipment is recorded as detected. If no corresponding safety protection equipment detection frame exists, and the personnel detection frame is not obstructed by personnel or touches the boundary of the preprocessed image frame, the status of that type of safety protection equipment is recorded as not detected.
[0241] If the overlap ratio between personnel detection frames exceeds the personnel occlusion judgment threshold, or if a personnel detection frame touches the boundary of a preprocessed image frame, the status of the safety protection equipment corresponding to the occluded or truncated location is recorded as "undetermined". The safety helmet corresponds to the upper position of the personnel detection frame, the safety belt and protective clothing correspond to the torso position of the personnel detection frame, and the safety protection equipment corresponds to the location range determined in step two based on the manual marking results. Only when the occluded area or the truncated area intersects with the location range of the corresponding safety protection equipment is the status of that type of safety protection equipment recorded as "undetermined".
[0242] The threshold for determining person occlusion is established using verification videos with manually marked occlusion locations. Multiple candidate thresholds for person occlusion are set. When the overlap ratio of two person detection boxes exceeds the candidate threshold, the two person detection boxes are considered to form an overlapping region.
[0243] For safety helmets, check whether the overlapping area intersects with the corresponding safety protection equipment position range at the top of the personnel detection frame; for safety belts and protective clothing, check whether the overlapping area intersects with the safety protection equipment position range corresponding to the torso position of the personnel detection frame; for safety protection equipment, check whether the overlapping area intersects with the corresponding position range determined in step two.
[0244] The number of unobstructed safety protection equipment locations that were incorrectly identified as being obstructed by people, and the number of safety protection equipment locations that were actually obstructed by people but were not identified as being obstructed by people, are counted. The candidate threshold for personnel obstruction with the smallest sum of the two is selected as the personnel obstruction judgment threshold.
[0245] When a person detection box touches the boundary of a preprocessed image frame, it means that the left boundary x-coordinate is equal to the lower limit of the preprocessed image frame x-coordinate, the right boundary x-coordinate is equal to the upper limit of the preprocessed image frame x-coordinate, and the upper boundary y-coordinate is equal to the lower limit of the preprocessed image frame y-coordinate or the lower boundary y-coordinate is equal to the upper limit of the preprocessed image frame y-coordinate.
[0246] Starting with the current preprocessed image frame as the ending position, multiple consecutive preprocessed image frames are obtained sequentially backward according to the frame number to form a wearing status judgment time period. The number of image frames included in the wearing status judgment time period is determined based on the frame rate of the high-risk operation video and the wearing status judgment duration. The wearing status judgment duration is determined using verification video that includes cases of wearing, not wearing, short-term occlusion, and short-term missed detection.
[0247] Multiple candidate wearing status judgment durations are set. The number of times a wearing status is incorrectly judged as not wearing, the number of times a not wearing status is incorrectly judged as wearing, and the delay time of the status judgment relative to the actual occurrence time are counted for each candidate wearing status judgment duration. First, the candidate wearing status judgment duration with the smallest sum of the first two types of errors is selected. If multiple candidate wearing status judgment durations have the same sum of error counts, the candidate wearing status judgment duration with the shortest delay time is selected.
[0248] During the period for determining the wearing status, preprocessed image frames with the safety protective equipment status set to "uncertain" are deleted, and the number of image frames with the status "detected" and "undetected" among the remaining preprocessed image frames is counted. If the number of remaining preprocessed image frames is less than the minimum number of valid image frames, the actual wearing status of that type of safety protective equipment for the person is recorded as "uncertain".
[0249] The minimum number of valid image frames is determined using a verification video. Multiple candidate minimum number of valid image frames are set, and the number of incorrect judgments due to insufficient valid image frames and the number of uncertain states due to excessive required image frames are counted. The candidate minimum number of valid image frames with the smallest sum of these two counts is selected.
[0250] When the number of remaining preprocessed image frames is greater than or equal to the minimum number of valid image frames, the state is calculated as the ratio of the number of detected image frames to the number of remaining preprocessed image frames. When this ratio is greater than or equal to the wearing confirmation ratio threshold, the actual wearing state is recorded as "wearing"; when this ratio is less than or equal to the non-wearing confirmation ratio threshold, the actual wearing state is recorded as "not wearing"; when this ratio is between the wearing confirmation ratio threshold and the non-wearing confirmation ratio threshold, the actual wearing state is recorded as "undetermined".
[0251] The wearing confirmation ratio threshold and the non-wearing confirmation ratio threshold are determined using verification videos. Multiple candidate wearing confirmation ratio thresholds and candidate non-wearing confirmation ratio thresholds are set, ensuring that the candidate wearing confirmation ratio threshold is greater than the candidate non-wearing confirmation ratio threshold. The number of incorrect judgments for the wearing, non-wearing, and uncertain states under each parameter combination is counted, and the parameter combination with the smallest sum of incorrect judgments is selected.
[0252] Save the personnel role, personnel detection frame, verified safety protection equipment detection frame, safety protection equipment category, detection confidence level, and actual wearing status corresponding to each personnel number, and store them in correspondence with the corresponding job number, job type, job area boundary, camera equipment number, timestamp, and frame sequence number.
[0253] In step four, the personnel number, personnel role, personnel detection frame, verified safety protection equipment detection frame, safety protection equipment category, detection confidence level, and actual wearing status saved in step three are read, along with the corresponding job number, job type, job area boundary, camera equipment number, timestamp, and frame sequence number. Simultaneously, the required safety protection equipment category saved in step one is read.
[0254] Data from the same high-risk operation video is arranged according to camera device number, timestamp, and frame number. Data with the same camera device number are arranged from earliest to latest timestamp; if timestamps are the same, they are arranged from smallest to largest frame number. Data with the same personnel number are arranged in the above order to form the continuous detection results for that personnel in the high-risk operation video.
[0255] The system determines whether a person is within the work area based on the personnel detection bounding box. The midpoint of the lower boundary of the personnel detection box is taken as the personnel location point. The x-coordinate of the personnel location point is the average of the x-coordinates of the left and right boundaries of the personnel detection box, and the y-coordinate is the y-coordinate of the lower boundary of the personnel detection box. The personnel location point is used to indicate the position of the person in contact with the ground or work platform within the preprocessed image frame.
[0256] Determine if the personnel's location is within the work area boundary determined in step one. To do this, draw a ray horizontally from the personnel's location to the right side of the preprocessed image frame and count the number of intersections between this ray and each line segment of the work area boundary. If the number of intersections is odd, the personnel's location is determined to be within the work area; if the number of intersections is even, the personnel's location is determined to be outside the work area. If the personnel's location is on a line segment of the work area boundary, it is treated as if it were within the work area.
[0257] When counting the number of intersections between a ray and the boundary segments of the work area, horizontal boundary segments are not included in the intersection count. When a ray passes through a common boundary point of two adjacent boundary segments, an intersection is counted only if one adjacent boundary segment at that common boundary point is above the ray and the other adjacent boundary segment is below the ray, to avoid the same common boundary point being counted repeatedly.
[0258] To determine whether a personnel location is on the boundary segment of the work area, check if the three points formed by the personnel location and the two endpoints of the boundary segment are collinear, and check if the x-coordinate and y-coordinate of the personnel location are within the corresponding coordinate ranges of the two endpoints of the boundary segment. If the above conditions are met, the personnel location is determined to be on the boundary segment of the work area.
[0259] When a person's location is within the work area, the actual wearing status of their personal protective equipment (PPE) is assessed. When a person's location is outside the work area, their actual wearing status of PPE at the current timestamp is not included in the PPE wearing results for this high-risk operation.
[0260] For each person located within the work area, sequentially read the required safety protection equipment categories saved in step one, and query the actual wearing status corresponding to the same personnel number, same timestamp, and same safety protection equipment category saved in step three.
[0261] The assessment result of a person regarding a required type of safety protective equipment at a specific timestamp is recorded as the wearing result of that type of safety protective equipment. The wearing result includes compliant, non-compliant, and indeterminate.
[0262] If the actual wearing status is "wearing", the wearing result of the corresponding safety protective equipment is determined to meet the requirements. If the actual wearing status is "not wearing", the wearing result of the corresponding safety protective equipment is determined to not meet the requirements. If the actual wearing status cannot be determined, the wearing result of the corresponding safety protective equipment is determined to be "undeterminable".
[0263] Step 3: If the actual wearing status of a person's safety protective equipment category at the current timestamp is not saved, check if a personnel detection frame exists for that person at the current timestamp. If a personnel detection frame exists, and the frame is not obstructed by a person or touches the boundary of the preprocessed image frame, the wearing result of the corresponding safety protective equipment is determined to be non-compliant. If a personnel detection frame exists, but is obstructed by a person or touches the boundary of the preprocessed image frame, the wearing result of the corresponding safety protective equipment is determined to be undeterminable. If no personnel detection frame exists for that person at the current timestamp, no wearing result is set for that timestamp.
[0264] For other safety protective equipment not listed in the category of safety protective equipment required to be worn in Step 1, only the test results are saved. The actual wearing status of the safety protective equipment is not used to determine whether the person meets the safety protective equipment wearing requirements for this high-risk operation.
[0265] The judgment result obtained by a person regarding all required safety protective equipment at a specific timestamp is recorded as the person's wearing result at that timestamp. If the wearing result of any required safety protective equipment is non-compliant, the person's wearing result is determined to be non-compliant. If the wearing results of all required safety protective equipment are compliant, the person's wearing result is determined to be compliant. If there are no non-compliant wearing results, but the wearing result of at least one required safety protective equipment is indeterminate, the person's wearing result is determined to be indeterminate.
[0266] If a person's role is uncertain, the assessment of their wearing of personal protective equipment (PPE) will still be based on the category of PPE required in Step 1. The assessment of PPE wearing will not be stopped simply because the person's role is uncertain. The person's role is only used to record their responsibilities in high-risk operations and is not used to replace the actual wearing status of PPE.
[0267] For each personnel number and each category of required safety protective equipment, the consecutive wearing results are checked according to timestamps. When the wearing result at a certain timestamp changes from compliant or uncertain to non-compliant, that timestamp is taken as the start time of a continuous non-wearing period. The continuous non-wearing period is the time range within which the same person continuously fails to meet the requirements for the same safety protective equipment.
[0268] If, after the start of a continuous period of non-wearing, the wearing result at subsequent timestamps still indicates non-compliance, the continuous period of non-wearing will be extended. When the wearing result at subsequent timestamps changes to meet the requirements, the last non-compliant timestamp before meeting the requirements will be used as the end time of the continuous period of non-wearing.
[0269] When a timestamp indicating an indeterminate wearing result occurs within a consecutive period of non-wearing, the number of consecutive indeterminate image frames is counted. If the number of consecutive indeterminate image frames does not exceed the allowable number of consecutive missing image frames determined in step three, and if a non-compliant wearing result occurs again after an indeterminate result, the non-compliant wearing results before and after the indeterminate result are merged into the same consecutive period of non-wearing.
[0270] If the number of consecutively undetermined image frames exceeds the allowable number of consecutively missing image frames determined in step three, the last non-compliant timestamp before the undetermined period will be used as the end time of the current consecutive unworn period. If another non-compliant period occurs after the undetermined period, a new consecutive unworn period will be established starting from the timestamp of the second non-compliant period.
[0271] When a continuous record corresponding to a certain personnel number terminates, if that personnel has an unfinished continuous period of not wearing a mask, the last timestamp of the personnel that did not meet the requirements will be used as the end time of the continuous period of not wearing a mask. When a personnel's location changes from within the work area to outside the work area, the continuous period of not wearing a mask will also end with the timestamp of the last time the personnel were within the work area and the wearing result was not in compliance with the requirements.
[0272] For each consecutive period of non-wearing, the time difference between the end time and the start time is calculated to obtain the duration of non-wearing. When the start and end times are the same, the duration of non-wearing is calculated based on the time interval between adjacent image frames in the high-risk operation video. The time interval between adjacent image frames is determined based on the frame rate saved in step one, i.e., the time obtained by dividing one by the frame rate.
[0273] Read the work order, work plan, or currently implemented safety operating procedures to determine whether the corresponding safety protective equipment is required to be worn continuously during the work period. If the safety protective equipment is specified to be worn continuously during the work period, any continuous period of non-wearing will be designated as a period of non-compliance with the safety protective equipment wearing requirements, and no further allowable non-wearing time will be set.
[0274] For each time period in which safety protective equipment is not worn as required, a representative image frame is selected from the pre-processed image frames contained within that time period. The representative image frame is a pre-processed image frame used to display the situation where the corresponding person is not wearing safety protective equipment.
[0275] When selecting a representative image frame, first exclude preprocessed image frames where the person detection bounding box is obscured by a person or touches the boundary of the preprocessed image frame. Among the remaining preprocessed image frames, select the preprocessed image frame with the highest confidence score for the person detection bounding box as the representative image frame. If multiple preprocessed image frames have the same confidence score for the person detection bounding box, select the preprocessed image frame with the earliest timestamp.
[0276] If, during the period when safety protective equipment is not worn as required, all preprocessed image frames show personnel occlusion or contact with the boundaries of the preprocessed image frames, the preprocessed image frame with the highest personnel detection confidence score is selected as the representative image frame, without excluding the aforementioned preprocessed image frames. If the detection confidence scores are the same, the preprocessed image frame with the earliest timestamp is selected.
[0277] Based on the camera device number and timestamp corresponding to the representative image frame, the original image frame with the same timestamp is read from the high-risk operation video captured in step one. If there is no original image frame with the exact same timestamp as the representative image frame in the high-risk operation video, the original image frame with the smallest timestamp difference is selected; if there are two original image frames with the same timestamp difference, the original image frame with the earlier timestamp is selected.
[0278] Copy the selected original image frame and mark the corresponding personnel detection boxes in the copied image to form an annotated image. The annotated image retains the content of the original image and adds personnel detection boxes, personnel numbers, personnel roles, safety protection equipment categories, and wearing result text. The original image frame remains unchanged; no detection boxes or text are directly added to the original image frame.
[0279] In the annotated image, the position of the personnel detection box uses the coordinates of the corresponding personnel detection box saved in step three. Next to the personnel detection box, write the corresponding personnel number and role, along with the category of unworn safety protective equipment. If the personnel role is uncertain, write "uncertain" according to the personnel role saved in step three, without replacing it with another personnel role.
[0280] For each period of time when safety protective equipment was not worn as required, a detection record should be established. The detection record should at least include the job number, job type, job area boundary, camera equipment number, personnel number, personnel role, safety protective equipment category, start time, end time, duration of non-wearing, personnel detection frame coordinates, personnel detection frame detection confidence level, original image frame, labeled image, and corresponding frame number.
[0281] When the same person has multiple safety protection equipment wearing results that do not meet the requirements within the same time range, a test record shall be established for each type of safety protection equipment. The same personnel number, personnel role, camera equipment number within the start time range, and operation number shall be used in each test record. Different safety protection equipment categories shall not be merged into a test result of an equipment category that is not clearly defined.
[0282] For consecutive time ranges where the wearing result is uncertain, separate "Uncertainty Records" are created according to personnel number and safety protection equipment category. Each "Uncertainty Record" stores the start and end times, corresponding personnel detection frame, personnel role, camera equipment number, job number, and the reason for the uncertainty. The reason for uncertainty is recorded according to the judgment results in step three: personnel obstruction, image boundary truncation, insufficient number of effective image frames, or the wearing status judgment ratio falling between the wearing confirmation ratio threshold and the non-wearing confirmation ratio threshold.
[0283] Summarize the wearing results of all personnel within the work time range according to the work order number. If any inspection record shows that the wearing of safety protective equipment is not in compliance with requirements within the work time range, the safety protective equipment wearing result corresponding to that work order number is determined to be non-compliant. If there are no inspection records showing that the wearing of safety protective equipment is non-compliant, but there are records that cannot be determined, the safety protective equipment wearing result corresponding to that work order number is determined to be indeterminate. If there are neither inspection records showing that the wearing of safety protective equipment is non-compliant nor records that cannot be determined, the safety protective equipment wearing result corresponding to that work order number is determined to be compliant.
[0284] In one possible embodiment, the high-risk operation video has an image width of 1,920 pixels, an image height of 1,080 pixels, and a frame rate of 25 frames per second. The YOLOv8 object detection algorithm uses an image input size of 640 pixels by 640 pixels, and personnel, safety helmets, safety belts, protective clothing, and safety protective equipment are each assigned a detection confidence threshold determined based on the validation image set.
[0285] In this embodiment, the initial length of the review observation segment is set to fifty preprocessed image frames, the temporary corresponding overlap boundary is set to 0.2, the temporary corresponding movement boundary is set to 0.25, the allowable range of height change is set to 0 to 0.3, the personnel overlap reference value is set to 0.15, the reference number is set to eight reference image frames, and the predetermined quantile position is set to the tenth and ninetieth quantile positions.
[0286] The allowable ranges for lateral and longitudinal motion are set to 0.08 times the width and height of the corresponding personnel detection frame, respectively. The position jump boundary is set to 0.2, and the number of confirmation frames for opposite movement direction is set to three pre-processed image frames. The visible confirmation boundary is set to 0.6, the width ratio of the surrounding area is set to 0.1, the number of confirmation frames for stable imaging state is set to five pre-processed image frames, and the number of synchronous confirmation frames is set to two pre-processed image frames.
[0287] The attribution advantage boundary is set to 0.15, the attribution handover confirmation count is set to three, the boundary proximity ratio is set to 0.05, and the maximum observation length is set to 150 preprocessed image frames. The initial missing confirmation length is determined based on the median number of missing frames of the same security equipment category in the verification video under short-term occlusion conditions.
[0288] The small size judgment threshold is set to 0.01% of the area of the safety protection equipment detection box to the area of the corresponding personnel detection box. The Faster R-CNN object detection algorithm uses a ResNet50 network as the feature extraction network, and the image region is adjusted to 800 pixels by 800 pixels while maintaining the aspect ratio. The insufficient horizontal and vertical parts are filled with image regions with a pixel value of zero.
[0289] The above values are used to illustrate that each parameter can be obtained and used in accordance with the determination method described in this embodiment, and do not limit this application to the specific values mentioned above. Under other camera equipment, image width, image height, frame rate, working area, or safety protection equipment categories, the corresponding parameters are re-determined according to the parameter determination image, training image group, verification image group, and verification video.
[0290] The results of wearing safety protective equipment corresponding to the job number are stored in relation to all inspection records and undetermined records under that job number. Inspection records are arranged from earliest to latest according to start time; if the start times are the same, they are arranged according to personnel number; if the personnel numbers are the same, they are arranged according to safety protective equipment category.
[0291] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.
[0292] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.
[0293] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0294] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0295] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0296] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for identifying and detecting safety protection equipment in high-risk operation videos of power plants, characterized in that, include: Obtain high-risk operation videos and the categories of required safety protective equipment, and perform image preprocessing on the high-risk operation videos to obtain preprocessed image frames; Target detection is performed on the preprocessed image frames to obtain personnel detection boxes and safety protection equipment detection boxes; Based on the position of the safety protection equipment detection frame relative to the personnel detection frame, the safety protection equipment detection frame and the personnel detection frame are matched to obtain the corresponding results; Based on the corresponding results and the required category of safety protective equipment, determine the safety protective equipment wearing result for each personnel detection frame, and generate a detection record of non-compliance with safety protective equipment wearing requirements based on the continuous safety protective equipment wearing results.
2. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 1, characterized in that: The step of performing image preprocessing on the high-risk operation video to obtain preprocessed image frames includes: Determine the boundary of the work area in each image frame of the high-risk operation video, retain the pixels within the boundary of the work area, and obtain the work area image; The image of the work area is sequentially subjected to bilateral filtering and desharpening to obtain an edge-enhanced image; The edge-enhanced image is sequentially subjected to contrast-limited adaptive histogram equalization and gamma correction to obtain the gamma-corrected image. Based on the proportion of bright pixels in the gamma-corrected image, the pixels exceeding the brightness threshold are compressed to obtain the preprocessed image frame.
3. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 1, characterized in that: The step of performing target detection on the preprocessed image frame to obtain personnel detection boxes and safety protection equipment detection boxes includes: The preprocessed image frame is input into the trained YOLOv8 object detection algorithm to obtain candidate detection boxes with category and detection confidence. Delete candidate detection boxes whose detection confidence is lower than the detection confidence threshold of the corresponding category; For multiple candidate detection boxes of the same category that overlap with each other, the candidate detection boxes with higher detection confidence are retained according to the overlap ratio between the candidate detection boxes. Candidate detection boxes categorized as "personnel" are identified as the "personnel detection boxes," and candidate detection boxes categorized as "safety protection equipment" are identified as the "safety protection equipment detection boxes." 4. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 1, characterized in that: Based on the personnel role identification regulations implemented on-site, determine the color of the work vest corresponding to each personnel role; The image within each personnel detection frame is converted into three color components: hue, saturation, and brightness. Based on the preset hue range, saturation lower limit, and brightness range, the work vest pixels corresponding to each work vest color are extracted. The work vest pixels are subjected to erosion and dilation processing and connected region filtering to obtain the work vest region corresponding to each work vest color, and the proportion of the number of pixels in each work vest region to the number of pixels in the personnel detection box is calculated. The ratios are compared with preset work vest area ratio thresholds, and the personnel role corresponding to the personnel detection box is determined based on the comparison results.
5. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 1, characterized in that: The step of associating the safety protection equipment detection frame with the personnel detection frame based on the position of the safety protection equipment detection frame relative to the personnel detection frame includes: Based on the position of various safety protection equipment relative to personnel in the marked images, determine the location range of each type of safety protection equipment; Calculate the lateral and longitudinal relative positions of the center point of each of the safety protection equipment detection frames within each of the personnel detection frames; A safety protection equipment detection frame whose center point is located within the personnel detection frame, and whose horizontal and vertical relative positions are within the location range of the corresponding category of safety protection equipment, corresponds to the personnel detection frame. When the same safety protection equipment detection frame meets the corresponding conditions of multiple personnel detection frames, the ratio of the intersection area between the safety protection equipment detection frame and each of the personnel detection frames is calculated, and it is matched with the personnel detection frame with the largest intersection area ratio. When the intersection area ratios are the same, it is matched with the personnel detection frame with the closest center point.
6. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 1, characterized in that: Before determining the safety protection equipment wearing result corresponding to each of the personnel detection frames, the method further includes: The review observation segments are obtained from consecutive preprocessed image frames in chronological order, and the personnel detection boxes in adjacent preprocessed image frames are temporarily matched to form a temporary personnel sequence. For each type of safety protection equipment required to be worn in the temporary personnel sequence, a reference image frame is selected from the review observation segment where the personnel detection frame does not touch the image boundary, the overlap ratio with other personnel detection frames is not greater than the preset personnel overlap reference value, and the safety protection equipment detection frame belongs to a unique entity. Based on the detection confidence of the safety protection equipment detection frame in the reference image frame, the ratio of the detection frame area to the corresponding personnel detection frame area, and the relative position of the center point, the confidence change interval, scale change interval, and position activity interval are determined, and the confidence change interval, scale change interval, and position activity interval are updated according to the newly added reference image frame. Determine the following relationship between the current safety protection equipment detection frame and the corresponding personnel detection frame, as well as the imaging state of the current preprocessed image frame. Based on the following relationship, imaging state, and comparison results between the current safety protection equipment detection frame and the confidence change interval, scale change interval, and position activity interval, determine the review status and generate a review object based on the review status.
7. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 6, characterized in that: Determining the tracking relationship between the current safety protection equipment detection frame and the corresponding personnel detection frame includes: Calculate the lateral and longitudinal movement distances of the center points of the safety protection equipment detection frames and the corresponding personnel detection frames in adjacent preprocessed image frames, respectively. Subtract the lateral and longitudinal movement distances of the center point of the personnel detection frame from the lateral and longitudinal movement distances of the center point of the safety protection equipment detection frame to obtain the lateral follow-up difference and the longitudinal follow-up difference. Then, normalize the lateral follow-up difference and the longitudinal follow-up difference according to the width and height of the previous personnel detection frame. The median values of the lateral and longitudinal motion differences within the reviewed observation segment are obtained respectively, and the deviation of the current lateral and longitudinal motion differences from the corresponding median values is calculated. When the deviation is within the allowable range of lateral and longitudinal movement respectively, and the safety protection equipment detection frame does not change position and does not continuously reach the number of confirmation frames in the opposite direction of movement, the safety protection equipment detection frame is determined as a detection result that moves with the personnel; otherwise, it is determined as a non-following detection result.
8. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 6, characterized in that: Determining the imaging state of the current preprocessed image frame includes: Calculate the average brightness difference and the change in the proportion of bright pixels within the corresponding person detection boxes in adjacent preprocessed image frames, and calculate the change in the standard deviation of pixel brightness in a predetermined width region around the person detection box; Based on the average brightness difference, the change in the proportion of bright pixels, and the change in the standard deviation of pixel brightness in the reference image frame, the corresponding reference change range is determined respectively; when any change exceeds the corresponding reference change range, the current preprocessed image frame is determined as the imaging state change frame; when all changes in a consecutive preset number of preprocessed image frames are within the corresponding reference change range, the corresponding preprocessed image frame is determined as the imaging state stable frame. An anomaly is defined as the first time the detection confidence level falls below the lower boundary of the confidence change interval, the first time the ratio of the detection frame area to the corresponding personnel detection frame area falls below the lower boundary of the scale change interval, or the first time the safety protection equipment detection frame partially disappears. When the difference in frame number between the detected anomaly and the frame of the change in imaging state is not greater than the number of synchronous confirmation frames, and the detection result of the safety protection equipment recovers to the corresponding change range in the subsequent frame of stable imaging state, the detected anomaly is determined to be a temporary imaging fluctuation; otherwise, the detected anomaly is determined to be a persistent anomaly.
9. The method for identifying and detecting safety protection equipment for high-risk operation videos in power plants according to claim 6, characterized in that: The step of determining the review status and generating a review object based on the review status includes: Based on the location range of the corresponding category of safety protection equipment, an area where the equipment should appear is established within each of the personnel detection frames. The portion of the area where the equipment should appear that is not covered by other personnel detection frames and does not exceed the boundary of the preprocessed image frame is determined as an effective visible area. Based on the proportion of the effective visible area to the area where the equipment should appear, it is determined whether the area where the equipment should appear is in a visible state or a restricted state. When the required safety protective equipment is not detected, if the area where the equipment should appear is in a restricted state, the verification observation segment is extended, and if it is still in a restricted state after reaching the maximum observation length, it is determined that the visibility condition is insufficient; if the area where the equipment should appear is in a visible state and the corresponding preprocessed image frame is an imaging state stable frame, it is determined that the stable state is missing when the number of consecutive undetected image frames reaches the current missing confirmation length. When the same safety protection equipment detection frame meets the corresponding conditions of multiple personnel detection frames, the corresponding relationship is re-determined based on the intersection area ratio and follow-up relationship between the safety protection equipment detection frame and each personnel detection frame. If the corresponding relationship cannot be uniquely determined, it is determined to be a conflict of ownership. Based on persistent anomalies, location anomalies, insufficient visibility conditions, missing stable states, or ownership conflicts, corresponding verification image regions are formed. These verification image regions are then input into the Faster R-CNN object detection algorithm, and the verification object is generated based on the verification detection results.
