A foreign matter out-of-boundary detection method, computer device and storage medium

Abstract

The application belongs to the field of industrial safety monitoring, and discloses a foreign matter border crossing detection method, computer equipment and a storage medium. The method comprises the following steps: acquiring a current video frame; inputting the current video frame into a warning line segmentation model and a foreign matter detection model respectively to determine a binary mask of the current video frame and a dynamic detection frame for foreign matter; determining at least one warning boundary point set of the current video frame based on the binary mask; performing linear regression fitting on each warning boundary point set to obtain a corresponding candidate fitting warning line, and generating a corresponding candidate warning area based on each candidate fitting warning line; selecting a target warning area with the highest overlap degree with a current calibration warning area from each candidate warning area; if the overlap degree is less than a preset threshold, regarding the target warning area as a new calibration warning area; and determining whether there is foreign matter border crossing in the current video frame based on the dynamic detection frame and the new calibration warning area. The application can improve the accuracy of foreign matter border crossing detection.

Description

Technical Field

[0001] This application relates to the field of industrial safety monitoring, specifically to a method for detecting foreign objects crossing boundaries, a computer device, and a storage medium. Background Technology

[0002] In industrial production processes, especially in automated operations such as metallurgy, stamping, conveying, and packaging, workers often need to perform auxiliary operations or inspections near equipment. However, because some processes involve high-speed moving mechanical parts, if personnel accidentally put their hands or other body parts into the operating area of ​​the equipment, it can easily lead to serious safety accidents, such as pinching, cutting, or other forms of personal injury. To avoid such risks, the industry generally adopts a variety of safety protection measures.

[0003] Currently, common practices mainly include physical isolation and inductive protection. Inductive protection commonly uses infrared beam detectors, laser scanning, or pressure sensors, which trigger alarms or shutdowns by setting a detection zone. However, these devices are easily affected by environmental factors such as dust, light, and vibration, leading to numerous false alarms and missed alarms. Furthermore, most methods cannot distinguish the type of object entering the hazardous area, failing to accurately determine whether it is a person's hands, clothing, or an unrelated object entering the detection range. Summary of the Invention

[0004] The purpose of this application is to provide a method, computer device, and storage medium for detecting foreign objects crossing boundaries, in order to solve the problem of insufficient accuracy in detecting foreign objects crossing the boundaries of industrial safety equipment in the prior art.

[0005] To achieve the above objectives, the first aspect of this application provides a method for detecting foreign object crossing boundaries, the method comprising:

[0006] Get the current video frame;

[0007] The current video frame is input into the warning line segmentation model and the foreign object detection model respectively to determine the binary mask of the current video frame and the dynamic detection box for the foreign object;

[0008] Determine at least one set of warning boundary points for the current video frame based on a binary mask;

[0009] Linear regression fitting is performed on each set of warning boundary points to obtain the corresponding candidate fitted warning line, and the corresponding candidate warning area is generated based on each candidate fitted warning line.

[0010] From all candidate warning areas, the candidate warning area with the highest overlap with the currently marked warning area is selected as the target warning area;

[0011] If the overlap between the target warning area and the current calibration warning area is less than a preset threshold, the target warning area will be used as a new calibration warning area for foreign object boundary detection in the next video frame.

[0012] Based on whether there is an intersection between the dynamic detection frame and the new calibration warning area, it is determined whether there is a foreign object crossing the boundary in the current video frame.

[0013] In this embodiment of the application, the step of determining at least one set of warning boundary points of the current video frame based on a binary mask includes: determining multiple candidate warning mask point sets of the current video frame based on a binary mask; determining the average vertical coordinate of all pixels in each candidate warning mask point set; and determining at least one set of warning boundary points based on the average vertical coordinate of each candidate warning mask point set and a preset vertical tolerance value.

[0014] In this embodiment, the candidate warning region has multiple first feature points, and the current calibration warning region has multiple second feature points. Each candidate warning region has multiple first feature points that correspond one-to-one with each of the multiple second feature points in the current calibration warning region. The spatial position of each first feature point in its candidate warning region is consistent with the spatial position of the corresponding second feature point in the current calibration warning region. The step of selecting the candidate warning region with the highest overlap with the current calibration warning region from among the candidate warning regions as the target warning region includes: constructing reference regions centered on each first feature point and each second feature point, with consistent shape and size; for any candidate warning region, summing the overlap areas of its reference regions with the corresponding reference regions of the current calibration warning region to determine the overlap reference parameter between any candidate warning region and the current calibration warning region; and based on the overlap reference parameter, selecting the candidate warning region with the highest overlap with the current calibration warning region from among the candidate warning regions as the target warning region.

[0015] In this embodiment of the application, multiple second feature points are arranged at equal intervals along the extension direction of the current calibration warning area inside or on its boundary; and / or, the reference area is a circular area with the first feature point or the second feature point as the center and a preset value as the radius.

[0016] In this embodiment, determining the overlap reference parameter between any candidate warning region and the currently calibrated warning region includes: summing the overlap areas of each circular region corresponding to the candidate warning region and the circular region corresponding to the currently calibrated warning region to obtain a total overlap area; using the total overlap area as the overlap reference parameter corresponding to the candidate warning region; or, using the ratio of the total overlap area to the total area of ​​all circular regions corresponding to the currently calibrated warning region as the overlap reference parameter corresponding to the candidate warning region; and selecting the target warning region from each candidate warning region based on the overlap reference parameter includes: selecting the candidate warning region with the largest overlap reference parameter as the target warning region.

[0017] In this embodiment, the step of generating corresponding candidate warning regions based on each candidate fitting warning line includes: determining the direction vector of the candidate fitting warning line according to the coordinate values ​​of the two endpoints of the candidate fitting warning line; determining the normal vector of the candidate fitting warning line based on the direction vector and the coordinate values ​​of the two endpoints of the candidate fitting warning line; the direction vector and the normal vector are calculated according to formula (1):

[0018] , (1)

[0019] in, It is the direction vector; It is the normal vector; The maximum coordinate value of the candidate fitted warning line on the x-axis; The minimum coordinate value of the candidate fitted warning line on the x-axis; The maximum coordinate value of the candidate fitted warning line on the y-axis; The minimum coordinate value of the candidate fitted warning line on the y-axis is used; the candidate warning area is determined based on the normal vector and the preset offset.

[0020] In this embodiment of the application, the method further includes: when the overlap between the target warning area and the current calibration warning area is greater than or equal to a preset threshold, maintaining the current calibration warning area as a new calibration warning area for foreign object boundary detection in the next video frame.

[0021] In this embodiment, determining whether a foreign object has crossed the boundary in the current video frame based on whether there is an intersection between the dynamic detection box and the new calibration warning area includes: if there is an intersection between the dynamic detection box and the new calibration warning area corresponding to each video frame, and the number of consecutive frames is greater than a preset number of frames, it is determined that a foreign object has crossed the boundary and an alarm is issued; at least one of the following is drawn on the corresponding video frame: the binary mask, the dynamic detection box, the new calibration warning area, and the candidate fitted warning line corresponding to the target warning area for each video frame used to determine that a foreign object has crossed the boundary, to generate a foreign object crossing the boundary image and upload it to the server.

[0022] A second aspect of this application provides a computer device, comprising: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and to implement the methods described above when executing the instructions.

[0023] A third aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the methods described above.

[0024] The above technical solution enables step-by-step abstraction and filtering of candidate warning regions from pixel-level segmentation results of warning lines to video frames. It also allows for dynamic updating based on overlap comparison with the calibrated warning region, ensuring the accuracy and stability of the warning region despite environmental changes and camera device offset. Furthermore, spatial intersection determination between the dynamic detection box and the updated calibrated warning region allows for precise judgment of whether foreign objects have crossed the boundary. This makes the entire boundary crossing detection process adaptive, robust, and reliable, effectively improving the accuracy of foreign object boundary crossing detection in industrial scenarios.

[0025] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0026] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0027] Figure 1 A flowchart illustrating a foreign object boundary crossing detection method according to an embodiment of this application is shown schematically.

[0028] Figure 2 A flowchart illustrating a foreign object boundary crossing detection method according to an embodiment of this application is shown schematically.

[0029] Figure 3 This illustration schematically shows the effect of foreign object boundary crossing detection according to an embodiment of this application;

[0030] Figure 4 This illustration schematically shows the effect of foreign object boundary crossing detection according to an embodiment of this application;

[0031] Figure 5 The diagram illustrates a structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0033] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0034] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0035] Figure 1 A flowchart illustrating a foreign object boundary crossing detection method according to an embodiment of this application is shown schematically. Figure 1 As shown in the figure, this application provides a method for detecting foreign objects crossing boundaries, which may include the following steps.

[0036] Step 101: Obtain the current video frame.

[0037] Step 102: Input the current video frame into the warning line segmentation model and the foreign object detection model respectively to determine the binary mask of the current video frame and the dynamic detection box for the foreign object.

[0038] In this embodiment, the current video frame is a single-frame color image acquired at the target time by an imaging device installed in the equipment's operating area; the warning line segmentation model is a semantic segmentation model that performs pixel-level recognition of the marking lines at the work site. Its output binary mask is a label map in the image coordinate system that marks whether or not the warning line belongs to the area, used to extract the boundary point set and geometric shape of the warning line; the foreign object detection model is a target detection model that estimates the position and category of objects that have entered or may enter the warning area. Foreign objects, under the higher-level concept, may include hands, tools, workpieces, clothing edges, and other entities related to safety risks. The dynamic detection box is a rectangular candidate box with position information and confidence level generated by the model on the current video frame, and can be updated over time in consecutive frames to reflect the temporal movement state of the object. By establishing regional and target information under a unified time reference, a reliable and robust input basis is provided for subsequent dynamic warning areas generated based on boundary fitting and normal offset, overlap determination based on spatial relationship measurement, and alarm logic linked to equipment operating status. Preferably, the warning line segmentation model can be based on the YOLOv8 model; the foreign object detection model can be used to detect human hands; the foreign object can be a human hand.

[0039] Step 103: Determine at least one set of warning boundary points for the current video frame based on the binary mask.

[0040] Step 104: Perform linear regression fitting on each set of warning boundary points to obtain the corresponding candidate fitted warning lines, and generate the corresponding candidate warning regions based on each candidate fitted warning line.

[0041] In this embodiment, the warning boundary point set is a set of discrete coordinates representing the outline of the warning line in the current video frame, belonging to structured data abstracted from pixel-level segmentation results. Based on this, a linear regression method is used to fit a straight-line model to each warning boundary point set. The resulting candidate fitted warning line is a geometric approximation of the actual warning line shape, providing a continuous, parameterized linear boundary expression in the image coordinate system. Further, the direction vector and unit normal vector are calculated based on the candidate fitted warning line, and a rectangular region is constructed according to a preset offset, thereby generating a candidate warning region. This candidate warning region serves as the spatial range established around the candidate fitted warning line, used for spatial relationship measurement with the foreign object detection box. This achieves the transformation from a pixel-level irregular red line mask to a geometrically measurable candidate warning region, enabling the warning line recognition result to participate in dynamic filtering, overlap rate calculation, and boundary violation determination in subsequent steps.

[0042] Step 105: Select the candidate warning area with the highest overlap with the currently marked warning area from all candidate warning areas, and use it as the target warning area.

[0043] Step 106: If the overlap between the target warning area and the current calibration warning area is less than a preset threshold, the target warning area is used as a new calibration warning area for foreign object boundary detection in the next video frame.

[0044] In this embodiment, spatial relationship measurements are performed between multiple candidate warning regions and the currently calibrated warning region to determine their overlap. The candidate warning region with the highest overlap is then selected as the target warning region. The candidate warning region is a spatial constraint area generated by candidate fitted warning lines and their normal extensions. The currently calibrated warning region is a reference region used as a judgment benchmark, obtained through manual calibration or historical updates. The target warning region is the final region used for boundary crossing judgment in the current frame after selection. Overlap, as a measure of spatial relationship, reflects the similarity between two regions in position and shape. When the overlap between the target warning region and the currently calibrated warning region is lower than a preset threshold, it is determined that the existing calibration can no longer accurately reflect the actual monitoring image. Therefore, the target warning region is updated to a new calibrated warning region, and this is used as a reference for foreign object boundary crossing detection in subsequent video frames. This achieves dynamic adaptive updating of the target warning region, maintaining the accuracy and stability of the judgment region even with slight camera device offsets or changes in environmental conditions, thereby ensuring the reliability of the subsequent boundary crossing detection process.

[0045] Step 107: Based on whether there is an intersection between the dynamic detection box and the new calibration warning area, determine whether there is a foreign object crossing the boundary in the current video frame.

[0046] In this embodiment, the dynamic detection box is a time-varying position representation of the intruder on the current video frame, and the new calibration warning area is a warning area used as the spatial reference for the current frame after prior screening and updating. The intersection of the two is used to measure the spatial overlap between the intruder and the warning area. When it is determined that there is an intersection, the output is that there is a foreign object crossing the boundary; otherwise, the output is that there is no foreign object crossing the boundary.

[0047] The above technical solution enables step-by-step abstraction and filtering of candidate warning regions from pixel-level segmentation results of warning lines to video frames. It also allows for dynamic updating based on overlap comparison with the calibrated warning region, ensuring the accuracy and stability of the warning region despite environmental changes and camera device offset. Furthermore, spatial intersection determination between the dynamic detection box and the updated calibrated warning region allows for precise judgment of whether foreign objects have crossed the boundary. This makes the entire boundary crossing detection process adaptive, robust, and reliable, effectively improving the accuracy of foreign object boundary crossing detection in industrial scenarios.

[0048] In this embodiment of the application, the step of determining at least one set of warning boundary points of the current video frame based on a binary mask includes: determining multiple candidate warning mask point sets of the current video frame based on a binary mask; determining the average vertical coordinate of all pixels in each candidate warning mask point set; and determining at least one set of warning boundary points based on the average vertical coordinate of each candidate warning mask point set and a preset vertical tolerance value.

[0049] In this embodiment, the candidate warning mask point set is used to represent a set of pixels generated by the segmentation result that may be spatially discontinuous but semantically belong to the same warning line; the average vertical coordinate is used to characterize the overall positional features of the same set in the vertical direction, thus serving as an aggregation criterion; the preset vertical tolerance value is the upper limit of the aggregation sensitivity, used to control the merging threshold when different sets are close in vertical position. Through the above processing, while maintaining semantic consistency, segments that are segmented due to occlusion, reflection, or noise can be merged into a structured warning boundary point set, reducing the risk of unstable fitting and misjudgment caused by fragmentation, and providing a stable, sparse, and computable input basis for subsequent linear regression fitting and candidate warning region generation.

[0050] In this embodiment, the candidate warning region has multiple first feature points, and the current calibration warning region has multiple second feature points. Each candidate warning region has multiple first feature points that correspond one-to-one with each of the multiple second feature points in the current calibration warning region. The spatial position of each first feature point in its candidate warning region is consistent with the spatial position of the corresponding second feature point in the current calibration warning region. The step of selecting the candidate warning region with the highest overlap with the current calibration warning region from among the candidate warning regions as the target warning region includes: constructing reference regions centered on each first feature point and each second feature point, with consistent shape and size; for any candidate warning region, summing the overlap areas of its reference regions with the corresponding reference regions of the current calibration warning region to determine the overlap reference parameter between any candidate warning region and the current calibration warning region; and based on the overlap reference parameter, selecting the candidate warning region with the highest overlap with the current calibration warning region from among the candidate warning regions as the target warning region.

[0051] In this embodiment, to achieve region filtering, a reference region with consistent shape and size is constructed centered on each first feature point and each second feature point. The reference region is a higher-level abstraction used to characterize the local spatial influence range around the feature point; it can be circular, elliptical, or other geometric shapes. In any candidate warning region, the overlap of its multiple reference regions with the reference region corresponding to the currently calibrated warning region is measured, and the overlap areas of each reference region are summed to obtain the overlap ratio reference parameter between the candidate warning region and the currently calibrated warning region. The overlap ratio reference parameter serves as an indicator to quantify the similarity between the candidate warning region and the currently calibrated warning region, measuring the degree of matching between different candidate warning regions. Based on this overlap ratio reference parameter, the candidate warning region with the highest overlap ratio with the currently calibrated warning region is selected from all candidate warning regions and ultimately determined as the target warning region. Based on this, when multiple candidate regions exist, a unified spatial metric can be used for screening to ensure that the selected target warning region is consistent with the current calibrated warning region in terms of geometric position and shape. This provides a stable reference region for subsequent boundary crossing detection and improves the adaptability and robustness of the overall solution under camera offset or fluctuations in on-site conditions, effectively improving the detection accuracy of foreign object boundary crossing in industrial scenarios.

[0052] In this embodiment of the application, multiple second feature points are arranged at equal intervals along the extension direction of the current calibration warning area inside or on its boundary; and / or, the reference area is a circular area with the first feature point or the second feature point as the center and a preset value as the radius.

[0053] In this embodiment, multiple second feature points are arranged at equal intervals within or on the boundary of the current calibration warning area, extending along its extension direction. These second feature points serve as spatial abstract identifiers of the calibration warning area, representing its baseline position distribution in geometric space. The extension direction represents the main geometric orientation of the warning area's shape, and their equal-interval arrangement ensures the representativeness and uniformity of the feature points within the area, thereby improving the reliability of subsequent spatial matching and measurement. Simultaneously, the baseline area is defined as a circular region centered on either the first or second feature point, with a preset value as its radius. The baseline area serves as a higher-level spatial extension of a single feature point, reflecting the influence domain of the feature point within a local area. The preset value, as a radius parameter, reflects the system's tolerance scale when determining spatial overlap within a local area. This expands discrete feature points into comparable baseline areas, and the regularized arrangement of second feature points achieves consistency in spatial structure between different warning areas. This provides a stable and measurable reference for subsequent region matching and filtering based on the overlap of the baseline area, effectively improving the detection accuracy of foreign object intrusion in industrial scenarios.

[0054] In this embodiment, determining the overlap reference parameter between any candidate warning region and the currently calibrated warning region includes: summing the overlap areas of each circular region corresponding to the candidate warning region and the circular region corresponding to the currently calibrated warning region to obtain a total overlap area; using the total overlap area as the overlap reference parameter corresponding to the candidate warning region; or, using the ratio of the total overlap area to the total area of ​​all circular regions corresponding to the currently calibrated warning region as the overlap reference parameter corresponding to the candidate warning region; and selecting the target warning region from each candidate warning region based on the overlap reference parameter includes: selecting the candidate warning region with the largest overlap reference parameter as the target warning region.

[0055] In this embodiment, the spatial relationship between any candidate warning region and the currently calibrated warning region is quantified to obtain an overlap benchmark parameter. Each circular region corresponding to the candidate warning region and the circular region corresponding to the currently calibrated warning region are constructed based on corresponding feature points to reflect the consistency of the local space. By summing the overlapping areas of these circular regions, a total overlapping area characterizing the overall spatial overlap can be obtained, which is used as the overlap benchmark parameter corresponding to the candidate warning region; alternatively, the ratio of this total overlapping area to the total area of ​​all circular regions corresponding to the currently calibrated warning region is used as the normalized overlap benchmark parameter, thereby improving comparability under different scale conditions. The overlap benchmark parameter, as a higher-level quantification index of spatial overlap, can reflect the degree of matching between the candidate warning region and the currently calibrated warning region in terms of geometry and position. Based on this, the region with the largest overlap benchmark parameter is selected from all candidate warning regions and determined as the target warning region. This allows for the selection of the optimal region using a unified metric when multiple candidate regions exist, ensuring that the target warning region is spatially most consistent with the currently calibrated warning region. This provides a stable and reliable spatial benchmark for subsequent foreign object boundary judgment, effectively improving the detection accuracy of foreign object boundary crossings in industrial scenarios.

[0056] In this embodiment, the step of generating corresponding candidate warning regions based on each candidate fitting warning line includes: determining the direction vector of the candidate fitting warning line according to the coordinate values ​​of the two endpoints of the candidate fitting warning line; determining the normal vector of the candidate fitting warning line based on the direction vector and the coordinate values ​​of the two endpoints of the candidate fitting warning line; the direction vector and the normal vector are calculated according to formula (1):

[0057] , (1)

[0058] in, It is the direction vector; It is the normal vector; The maximum coordinate value of the candidate fitted warning line on the x-axis; The minimum coordinate value of the candidate fitted warning line on the x-axis; The maximum coordinate value of the candidate fitted warning line on the y-axis; The minimum coordinate value of the candidate fitted warning line on the y-axis is used; the candidate warning area is determined based on the normal vector and the preset offset.

[0059] In this embodiment, candidate warning regions are generated based on each candidate fitted warning line to achieve the transformation from linear boundaries to spatial regions. The candidate fitted warning lines are obtained by linear regression of the warning boundary point set. The coordinate values ​​of their two endpoints are used to determine the direction vector, which serves as a geometric quantity representing the extension direction of the candidate fitted warning line and provides a benchmark for subsequent region construction. Based on this, the normal vector is further determined by combining the coordinate values ​​of the two endpoints. The normal vector, as a spatial geometric vector perpendicular to the direction vector, is used to define the expansion direction of the warning region. Then, the normal vector and a preset offset are used to generate candidate warning regions. Each candidate warning region is a rectangular area formed by extending in the direction of the normal vector with the candidate fitted warning line as its central boundary, representing a certain width range around the warning line in the actual working scenario. Through this step, candidate fitted warning lines with only one-dimensional geometric features can be upgraded to candidate warning regions with two-dimensional spatial coverage, transforming the warning line information into a spatial benchmark that can be used for subsequent overlap measurement and foreign object boundary judgment, effectively improving the detection accuracy of foreign object boundary crossings in industrial scenarios.

[0060] In this embodiment of the application, the method further includes: when the overlap between the target warning area and the current calibration warning area is greater than or equal to a preset threshold, maintaining the current calibration warning area as a new calibration warning area for foreign object boundary detection in the next video frame.

[0061] In this embodiment, a threshold is used to determine the overlap between the target warning area and the currently calibrated warning area. When the overlap is greater than or equal to a preset threshold, the target warning area and the currently calibrated warning area are considered to be consistent in spatial location and geometry. Therefore, there is no need to update the calibration results, and the current calibrated warning area is directly maintained as the new calibrated warning area and applied to the foreign object boundary crossing detection in the next video frame. The target warning area is the final reference area for judgment in this frame, obtained by screening candidate warning areas. The currently calibrated warning area is the spatial benchmark used for continuous tracking and judgment. The overlap is an important measure of spatial consistency between the two. The preset threshold is a tolerance range set during the judgment process to control the update sensitivity. This ensures the continuity and stability of the calibration area when the area matching degree is high enough, avoids jitter or misjudgment caused by frequent switching, ensures the consistency and reliability of boundary crossing detection judgment in the time series, and effectively improves the detection accuracy of foreign object boundary crossing in industrial scenarios.

[0062] In this embodiment, determining whether a foreign object has crossed the boundary in the current video frame based on whether there is an intersection between the dynamic detection box and the new calibration warning area includes: if there is an intersection between the dynamic detection box and the new calibration warning area corresponding to each video frame, and the number of consecutive frames is greater than a preset number of frames, it is determined that a foreign object has crossed the boundary and an alarm is issued; at least one of the following is drawn on the corresponding video frame: the binary mask, the dynamic detection box, the new calibration warning area, and the candidate fitted warning line corresponding to the target warning area for each video frame used to determine that a foreign object has crossed the boundary, to generate a foreign object crossing the boundary image and upload it to the server.

[0063] In this embodiment, boundary crossing is determined based on the spatial relationship between the dynamic detection box and the new calibration warning area. When the dynamic detection box and the new calibration warning area intersect, and the number of consecutive frames in the video frame where this intersection appears is greater than a preset number of frames, it is determined that a foreign object has crossed the boundary and an alarm is triggered. The dynamic detection box is a rectangular area representing the spatial location of the foreign object generated by the foreign object detection model in the current video frame. The new calibration warning area is a reference area for detection in this frame obtained under the action of the judgment and update mechanism. The preset number of frames is a constraint condition in the time dimension, used to avoid false alarms caused by single-frame jitter or occasional false detections. After confirming that a foreign object has crossed the boundary, the system will also overlay at least one of the key elements used to form the judgment basis, including the binary mask corresponding to the current video frame, the dynamic detection box, the new calibration warning area, and the candidate fitted warning line corresponding to the target warning area, onto the video frame image to generate a visualized foreign object boundary crossing image, and upload the image to the server. It not only achieves boundary crossing determination based on the combination of spatial intersection and temporal continuity, improving the accuracy and robustness of alarms, but also enables visualization and remote evidence storage of detection results, facilitating subsequent manual verification and system auditing, thereby enhancing the traceability and reliability of the overall security monitoring process.

[0064] The above technical solution enables a complete processing chain in industrial scenarios, from video frame acquisition, warning line segmentation, boundary point set extraction, linear regression fitting to candidate warning region generation. By constructing feature points, a reference region, and overlap benchmark parameters, it achieves quantitative comparison and dynamic filtering between candidate and calibrated warning regions, maintaining the accuracy and adaptability of the warning region even with slight camera device shifts, lighting changes, or environmental interference. Furthermore, combined with an overlap threshold determination mechanism, it maintains stability when the target warning region matches the currently calibrated warning region. When the difference between the two is too large, an update is performed to ensure the continuity and reliability of the warning area in the time dimension. Furthermore, based on the intersection of the dynamic detection box and the new calibrated warning area, combined with the time constraint of the preset number of frames, it is possible to effectively distinguish between occasional detection errors and real boundary crossing behavior, thereby improving the accuracy of alarms. Finally, when a boundary crossing is established, the binary mask, dynamic detection box, calibrated warning area and candidate fitted warning line are visualized, overlaid and uploaded to the server. This not only realizes the intuitive display of the results, but also provides traceable evidence storage capabilities, significantly improving the accuracy of foreign object boundary crossing detection in complex industrial scenarios.

[0065] The following is an example of this application:

[0066] Figure 2 A flowchart illustrating a foreign object boundary crossing detection method according to an embodiment of this application is shown schematically. Figure 2 As shown in this embodiment, firstly, images are acquired using a top camera to obtain the original video frames of the current work scene; then, the YOLOv8-Seg model is used for image segmentation, outputting a binary mask M to extract the pixel region of the red warning line; simultaneously, the YOLOv8 hand detection model is used for target detection, outputting a set of hand detection boxes. This is used to represent the location of foreign objects that may exist in the current frame; after obtaining the segmentation result, the red line mask is further extracted and fitted to obtain a linear straight line. This enables the geometric representation of the warning line; based on the direction vector and normal vector of the fitted line, and combined with a preset offset, a dynamic warning area is generated. This is used to construct a measurable danger zone; subsequently, robustness checks are performed, comparing the dynamically generated warning zone with the reference zone. The system performs comparisons to ensure stable region detection even under camera shake or environmental changes; during the boundary detection phase, the system checks the hand detection frame. With dynamic alert zone The system checks for spatial intersections; if intersections exist, it indicates that a foreign object has entered the danger zone. Finally, in the alarm triggering phase, the system issues an alarm signal and records alarm images and logs, creating a traceable safety event file. Through the coordinated action of these steps, the process achieves closed-loop processing from image input, warning line recognition, region generation, robustness correction to boundary crossing detection and alarm output, ensuring accurate identification and timely warning of personnel crossing boundaries in complex industrial environments.

[0067] 1. Image Acquisition and Model Loading:

[0068] Images are captured from a camera mounted directly above the device. And load two deep learning models: (1) warning line segmentation model based on YOLO architecture. Output binary mask (2) Foreign object detection model for detecting human hand targets The output is a series of detection boxes. .

[0069] 2. Warning line coordinate extraction and aggregation:

[0070] Due to interference from factors such as hands, equipment obstruction, and ground reflections, the segmentation model in the image... After the above reasoning, multiple separate sets of candidate warning mask points may be obtained. Each of them Indicates the first A set of candidate warning mask points for each connected region.

[0071] To merge multiple segments belonging to the same warning line, define each point set. Average ordinate:

[0072]

[0073] Setting longitudinal tolerance , will satisfy:

[0074] point set Aggregated into the same set of warning boundary points Finally obtained Group of warning boundary point sets Each group is considered as a candidate segment of the logical red line.

[0075] 3. Fit the warning line:

[0076] For each set of warning boundary points A linear regression model is used:

[0077]

[0078] Among them, parameters Obtained by minimizing the sum of squared residuals and Here are the x and y coordinates of any point on the fitted warning line after linear regression:

[0079] The result Fitted straight lines As a warning line for candidate fitting.

[0080] 4. Construction of candidate alert zones:

[0081] The two endpoints of each candidate fitting warning line are defined by the corresponding set of warning boundary points. Maximum and minimum The value is used to determine this. Let the two endpoints of the candidate fitting warning line be... Then the direction vector is:

[0082]

[0083] Along the normal vector Directional offset by a fixed distance Construct a rectangular region That is, the candidate warning area, which is composed of a set of points. constitute.

[0084] 5. Calibration warning zone offset detection mechanism:

[0085] To enhance system stability and reduce false alarms, the manually calibrated calibration warning area is set to a rectangle. Its lower boundary has three key points:

[0086]

[0087] For each candidate fitting warning line Calculate the three points corresponding to its lower boundary. And construct a circle with three pairs of points as centers and a radius of . Circular region pairs:

[0088]

[0089] Calculate their intersection area respectively The total area of ​​the standard circle is:

[0090]

[0091] The overlap rate between the candidate warning area and the designated warning area is defined as:

[0092]

[0093] Finally, the candidate with the highest overlap rate is selected from all candidate fitting warning lines. Its corresponding candidate fitting warning line The current video frame is considered the warning line, and its corresponding candidate warning area is taken as the target warning area.

[0094] If the maximum overlap rate is still less than the preset threshold ,Right now:

[0095]

[0096] If the warning line detection is found to be off-target, the target warning area box can be used instead of the calibrated warning area box for subsequent intersection determination.

[0097] 6. Hand detection and boundary crossing judgment:

[0098] For the detection frame Constructed rectangular area With the designated warning area Use a spatial geometry library (such as shapely) to determine if an intersection exists:

[0099]

[0100] Record the out-of-bounds status flag. Introduce a continuous frame mechanism; if consecutive frames... If the frame rate exceeds the limit, an alarm process is triggered.

[0101] 7. Equipment status linkage and alarm handling:

[0102] The system reads the current operating status of the device. Only The out-of-bounds detection logic is activated. If an alarm is triggered, the image is saved. The test results are collected and uploaded to the platform for subsequent analysis and auditing.

[0103] 8. Image synthesis and result output:

[0104] The necessary detection elements (segmentation mask, fitted lines, rectangular warning boxes, detection boxes, etc.) are drawn and overlaid using tools such as OpenCV. The output image supports local saving and remote transmission for human-computer interaction and security verification.

[0105] The following is another example:

[0106] In this embodiment, the system is deployed in the discharge area of ​​the sawing section of a metallurgical enterprise, a scenario with a high risk of accidental hand insertion into the equipment. A camera is installed directly above the discharge port to capture image frames.

[0107] First, the system loads the warning line segmentation model (custom-trained YOLOv8). After image input, a binary mask is obtained, and the boundary points of non-zero regions are extracted. Line segments are aggregated based on the difference in ordinates of the point sets, δy=50 pixels, and fitted to straight lines using the LinearRegression model in scikit-learn.

[0108] Subsequently, the direction of the line's normal vector is calculated, and a rectangular warning zone is constructed with a normal offset of 200 pixels. A red box generated using OpenCV is then overlaid on the image as the danger zone for the current frame.

[0109] To prevent camera shake from causing the manually calibrated warning bounding box to drift, the system introduces an offset detection mechanism. A manual standard bounding box is set, and its lower boundary three points are extracted. During the detection phase, a circle with a radius of 20 pixels is constructed for each point, and its overlap area is compared with the currently fitted bounding box. If the overlap rate is less than 10%, the system will indicate camera offset and use a newly identified warning bounding box for the next step.

[0110] Next, the current frame is input into the foreign object detection model to extract all detection boxes categorized as "hand," which are then converted into polygons and their intersection with the warning area is determined. If an intersection occurs and the device is running, the frame is recorded as out of bounds. The system employs a 3-frame consecutive detection mechanism; if three consecutive frames detect out of bounds, the current image is saved to the local static directory, and an alarm record is pushed to the security platform via API.

[0111] The final output image includes: a calibrated warning area, a dynamic detection box, and an out-of-bounds alarm text, which can be used for alarm, recording, and manual review.

[0112] Figure 3 The illustration shows the effect of foreign object boundary crossing detection according to an embodiment of this application. Figure 3 As shown, the large bounding box around the hand is the calibration warning region, and the smaller bounding boxes on the left and right sides of the hand represent the candidate fitting warning lines corresponding to the calibration warning region. The values ​​above the smaller bounding boxes are the model confidence scores; for example, line 0.93 and line 0.72 indicate the confidence scores of these two lines being identified as candidate fitting warning lines. We can see a hand reaching into the calibration warning region, but in this frame, the system has only identified the calibration warning region and the candidate fitting warning lines, and has not yet generated a bounding box for the hand. Therefore... Figure 3 It is mainly used to display segmentation and fitting results, and has not yet entered the out-of-bounds judgment stage.

[0113] Figure 4 The illustration shows the effect of foreign object boundary crossing detection according to an embodiment of this application. Figure 4As shown in the image, the bounding box closer to the hand is a dynamic detection bounding box identified by the foreign object detection model. The value Conf:0.95 above the box indicates the detection confidence, meaning the probability that the area contains a hand is 95%. The box further away is the designated warning area. The system, based on its judgment rules, found that the dynamic detection bounding box and the designated warning area overlapped, and the overlap lasted for more than a preset number of frames. Therefore, an alarm was triggered, and a notification was overlaid on the image.

[0114] Figure 5 A schematic diagram illustrating the structure of a computer device according to an embodiment of this application is provided. Figure 5 As shown in the illustration, this application provides a computer device that may include:

[0115] Memory 510 is configured to store instructions; and

[0116] Processor 520 is configured to retrieve instructions from memory 510 and to implement the methods described above when executing instructions.

[0117] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting foreign object crossing boundaries, characterized in that, The method includes: Get the current video frame; The current video frame is input into the warning line segmentation model and the foreign object detection model respectively to determine the binary mask of the current video frame and the dynamic detection box for the foreign object; Based on the binary mask, at least one set of warning boundary points for the current video frame is determined; Linear regression fitting is performed on each set of warning boundary points to obtain the corresponding candidate fitted warning line, and a corresponding candidate warning area is generated based on each candidate fitted warning line. The candidate warning region with the highest overlap with the current calibrated warning region is selected from each of the candidate warning regions and designated as the target warning region. Each candidate warning region has multiple first feature points, and the current calibrated warning region has multiple second feature points. Each of the multiple first feature points in each candidate warning region corresponds one-to-one with the multiple second feature points in the current calibrated warning region. The spatial position of each first feature point in its candidate warning region is consistent with the spatial position of the corresponding second feature point in its current calibrated warning region. The multiple second feature points are arranged at equal intervals along the extension direction of the current calibrated warning region, either inside or on its boundary. If the overlap between the target warning area and the current calibration warning area is less than a preset threshold, the target warning area will be used as a new calibration warning area for foreign object boundary detection in the next video frame. Based on whether there is an intersection between the dynamic detection frame and the new calibration warning area, it is determined whether the current video frame contains the foreign object crossing the boundary. The step of selecting the candidate warning region with the highest overlap with the currently calibrated warning region from among the candidate warning regions, and using it as the target warning region, includes: Construct reference regions with consistent shape and size, centered on each first feature point and each second feature point; the reference regions are circular regions with the first feature point or the second feature point as the center and a preset value as the radius; For any candidate warning area, the overlapping areas of each of its reference areas and the reference area corresponding to the current calibrated warning area are summed to determine the overlap reference parameter between any candidate warning area and the current calibrated warning area. Based on the aforementioned overlap reference parameters, the candidate warning region with the highest overlap with the currently calibrated warning region is selected from each of the candidate warning regions and used as the target warning region.

2. The method according to claim 1, characterized in that, The step of determining at least one set of warning boundary points for the current video frame based on the binary mask includes: Based on the binary mask, determine a set of multiple candidate warning mask points for the current video frame; Determine the average vertical coordinate of all pixels in each candidate warning mask point set; At least one set of warning boundary points is determined based on the average value of the longitudinal coordinates corresponding to each candidate warning mask point set and the preset longitudinal tolerance value.

3. The method according to claim 2, characterized in that, The benchmark parameters for determining the overlap between any candidate warning region and the currently calibrated warning region include: The total overlapping area is obtained by summing the overlapping areas of each circular area corresponding to the candidate warning area and the circular area corresponding to the currently calibrated warning area. The total overlapping area is used as the overlap benchmark parameter corresponding to the candidate warning area; Alternatively, the ratio of the total overlapping area to the total area of ​​all circular areas corresponding to the current calibrated warning area can be used as the overlap benchmark parameter for the candidate warning area. Based on the overlap reference parameter, the target warning region is selected from each of the candidate warning regions, including: The candidate warning region with the largest overlap reference parameter is taken as the target warning region.

4. The method according to any one of claims 1 to 3, characterized in that, The step of generating corresponding candidate warning regions based on each of the candidate fitted warning lines includes: The direction vector of the candidate fitted warning line is determined based on the coordinate values ​​of the two endpoints of the candidate fitted warning line. The normal vector of the candidate fitting warning line is determined based on the coordinates of the two endpoints of the direction vector and the candidate fitting warning line; the direction vector and the normal vector are calculated according to formula (1): ， （1） in, The direction vector; Let be the normal vector; The maximum coordinate value of the candidate fitting warning line on the x-axis; The minimum coordinate value of the candidate fitting warning line on the x-axis; The maximum coordinate value of the candidate fitted warning line on the y-axis; The minimum coordinate value of the candidate fitted warning line on the y-axis; The candidate warning area is determined based on the normal vector and the preset offset.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the overlap between the target warning area and the current calibration warning area is greater than or equal to the preset threshold, the current calibration warning area is retained as the new calibration warning area for foreign object boundary detection in the next video frame.

6. The method according to any one of claims 1 to 3, characterized in that, The step of determining whether the current video frame contains an intrusive object based on whether there is an intersection between the dynamic detection frame and the new calibration warning area includes: If there is an intersection between the dynamic detection frame and the new calibration warning area corresponding to each video frame, and the number of consecutive frames is greater than the preset number of frames, it is determined that a foreign object has crossed the boundary and an alarm is issued. At least one of the following is used to determine the presence of a foreign object crossing the boundary: the binary mask, the dynamic detection box, the new calibration warning area, and the candidate fitted warning line corresponding to the target warning area for each video frame. This is plotted on the corresponding video frame to generate a foreign object crossing the boundary image and upload it to the server.

7. A computer device, characterized in that, include: The memory is configured to store instructions; And a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method according to any one of claims 1 to 6.

8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method according to any one of claims 1 to 6.

Images