Monitoring and real-time alarming method and system for mobile manipulator operation safety

Abstract

The application provides a kind of mobile mechanical arm operation safety-oriented monitoring and real-time alarm method and system, comprising: step S1: based on real-time video stream frame image, target detection model is used to carry out target detection and positioning to mechanical arm and worker;Step S2: according to the position coordinates of mechanical arm, the center point coordinates are calculated;By comparing the pixel displacement of the center point of mechanical arm between consecutive video frames, it is judged whether mechanical arm is in motion state;Step S3: when mechanical arm is in motion state, then based on two-dimensional work area coordinate system, it is judged whether mechanical arm is in worker work area;Step S4: based on two-dimensional work area coordinate system, it is judged whether worker is in worker work area.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology, specifically to a monitoring and real-time alarm method and system for the safety of mobile robotic arm operations, and more specifically to a vision-based visual monitoring and real-time alarm method and system for the robotic arm operator's work area in a mobile state. Background Technology

[0002] Currently, existing technical solutions for personnel safety protection in mobile robotic arm operation scenarios have the following limitations: First, warning methods based on robotic arm projection lights can only project static markings directly below the robotic arm, failing to provide effective early warning for the robotic arm operator's work area during movement; Second, while monitoring systems relying on multi-sensor arrays possess certain dynamic perception capabilities, they suffer from inherent drawbacks such as high hardware costs, complex system integration, and difficulties in on-site calibration and maintenance, and are also unable to adapt to the complex changes in the robotic arm's movement trajectory in real time; Third, methods relying entirely on manual monitoring generally suffer from insufficient real-time performance, susceptibility to missed or misjudged cases due to personnel fatigue, and delayed safety responses.

[0003] Patent document CN120876833A (application number: 202511006300.5) discloses a lifting safety detection method and system based on artificial intelligence. By combining RGB images and depth information to construct panoramic perception and using trajectory modeling for risk prediction, it improves the target perception capability in complex scenarios to a certain extent. However, this solution has the following main limitations: First, it relies on expensive multimodal sensor arrays, resulting in high system hardware costs, complex deployment, and difficult maintenance; second, its technical path is complex, involving multi-source data fusion and trajectory prediction models, which face challenges such as high computational load, difficulty in ensuring real-time performance, and high requirements for model generalization in practical engineering applications; finally, as a general-purpose safety monitoring system, when dealing with the specific scenario of "real-time intrusion judgment of dynamic dangerous areas of mobile robotic arms," ​​which has extremely high requirements for real-time performance and positioning accuracy, its complex process may introduce unnecessary delays and uncertainties, and its cost-effectiveness is low.

[0004] Patent document CN120823685A (application number: 202511309045.1) discloses a tower crane operation safety early warning method based on multi-model fusion and dynamic threshold calculation. This scheme uses the Euclidean distance between personnel and the hook as the basic parameter for risk assessment, and combines it with other safety detection elements for comprehensive judgment and early warning output. While this scheme reflects a dynamic assessment approach to construction safety risks, its core logic, technical path, and application scenarios differ substantially from the solution in this patent, and therefore cannot be directly applied to the specific application needs addressed by this invention. Its main limitations are as follows: First, the early warning model of this scheme uses Euclidean distance as the core metric, and its early warning range mainly targets personnel approach risks, while this invention needs to monitor the dangerous areas involved in the entire movement path of the robotic arm, and its early warning range is dynamic and continuous; the two are fundamentally different in terms of spatial coverage logic. Second, the risk assessment system of this scheme is not well-matched with the actual working conditions faced by this invention. In the specific operational scenario involved in this invention, the objects lifted by the robotic arm are mainly heavy, regular loads such as steel coils and coils. Their motion trajectories are relatively stable, and the additional risks caused by instantaneous large swings are relatively low. Therefore, the complex correlation model constructed for dynamic parameters such as swing angle and speed in the comparative scheme has limited necessity and applicability in this scenario.

[0005] Patent document CN117819416A (application number: 202410015086.9) discloses a hoisting safety early warning system based on the fusion of machine vision and UWB, which achieves real-time monitoring and early warning of the boom position and safe distance through multi-sensor data fusion. Although this solution integrates multi-source information perception and fusion positioning, its technical approach differs significantly from that of this invention. This solution relies on the fusion of machine vision and UWB (ultra-wideband) and other multi-sensor technologies, resulting in a complex system structure, high hardware costs, and the need for professional calibration and synchronization during deployment and maintenance, which is not conducive to large-scale promotion or cost-sensitive industrial scenarios. Furthermore, this solution mainly targets collision avoidance warnings between cranes and stationary electrical equipment, and its risk model focuses on static distance threshold judgment. It lacks effective real-time modeling and adaptive early warning capabilities for dynamic and irregular safety zones formed by mobile robotic arms during continuous operation.

[0006] While the aforementioned existing technologies all introduce vision-based safety warning schemes, they are difficult to implement under complex conditions with a single camera, making personnel intrusion warnings challenging. This invention proposes an innovative, low-cost monocular vision monitoring solution for personnel intrusion warning scenarios involving mobile robotic arms. This method acquires video streams from the work site using a single camera and uses a deep learning object detection algorithm to locate the robotic arm and personnel targets in real time. Furthermore, perspective transformation technology is used to accurately map the two-dimensional image coordinates to the world coordinate system of the actual three-dimensional work space, thereby achieving precise spatial calibration of the robotic arm's position and the preset work area. This process incorporates the robotic arm's height information, effectively overcoming the inherent limitations of monocular vision in depth perception, achieving reliable judgment of spatial relationships, and laying the foundation for subsequent personnel intrusion detection. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a monitoring and real-time alarm method and system for the safe operation of mobile robotic arms.

[0008] A monitoring and real-time alarm method for mobile robotic arm operation safety provided by the present invention includes: Step S1: Based on real-time video stream frames, use a target detection model to perform target detection and localization on the robotic arm and worker; Step S2: Calculate the coordinates of the center point of the robotic arm based on its position coordinates; determine whether the robotic arm is in motion by comparing the pixel displacement of the center point of the robotic arm between consecutive video frames. Step S3: When the robotic arm is in motion, determine whether the robotic arm is within the worker's work area based on the two-dimensional work area coordinate system; Step S4: Determine whether the worker is within the worker's work area based on the two-dimensional work area coordinate system.

[0009] Preferably, step S1 includes: Step S1.1: Using fixed cameras deployed at the work site, collect original images covering the entire working area of ​​the robotic arm and typical activity areas of the worker that meet preset requirements under different lighting conditions and time periods; perform preliminary screening of the collected original images, removing invalid images that do not contain the robotic arm or worker, and ensuring that images of the robotic arm in different postures and spatial positions, as well as images of the worker in different postures and in different areas, all account for a certain proportion in the dataset; after screening, finally construct a set of effective images containing a preset number; Step S1.2: Use the labeling tool, including LabelMe, to label each image in the obtained valid image set; for each robotic arm instance and each worker instance appearing in the image, use a rectangular bounding box to label the complete target area; when labeling, use the coordinates of the upper left and lower right corners of the target's bounding rectangle as the reference, and store the labeling results in the YOLO standard format. Each piece of information consists of the target category index and the normalized bounding box coordinates, thereby completing the construction of the dataset; Step S1.3: Build an object detection model based on the YOLOv8s architecture, train the object detection model based on the built dataset, and obtain the trained object detection model; Step S1.4: Based on real-time video stream frames, use the trained target detection model to perform target detection and localization on the robotic arm and the worker.

[0010] Preferably, step S2 includes: when the displacement of the center point of the robotic arm position coordinates in two consecutive frames exceeds the pixel displacement threshold, the robotic arm is determined to be in a moving state;

[0011] Where t represents the number of frames. Let x be the center point of the robotic arm on the x-axis of frame t. Let y be the center point of the robotic arm on the t-th frame; Let x be the center point of the robotic arm on the x-axis of the (t-1)th frame; Let y be the center point of the robotic arm on the (t-1)th frame; if If the value exceeds the pixel displacement threshold, it is determined to be in a moving state; if If the value is less than or equal to the pixel displacement threshold, it is determined to be in a non-moving state.

[0012] Preferably, step S3 includes: Step S3.1: Identify and extract the coordinates of the worker's work area on the ground from the current view of the camera; define the two-dimensional coordinates of the current worker's work area on the real world ground from a top-down perspective; calculate the homogeneous coordinate transformation matrix based on the coordinates of the worker's work area in the original image and the two-dimensional coordinates of the worker's work area on the real world ground from a top-down perspective. The homogeneous coordinate transformation matrix is:

[0013] in, These are the original image coordinates. These are the transformed top-view plane coordinates, and s is a scaling factor; This is the homogeneous coordinate transformation matrix; Step S3.2: Based on the target region information of the robotic arm obtained by the target detection model, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Based on the homogeneous coordinate transformation matrix M, the point... Transform to the same two-dimensional coordinate system viewed from above to obtain its corresponding ground projection coordinates. This coordinate represents the horizontal center position of the robotic arm in the ground reference frame. Step S3.3: Obtain the two-dimensional ground projection point of the robotic arm. Then, the projected height of the robotic arm is calculated, and the area of ​​the projected region is calculated based on the projected height of the robotic arm and the pixel area. Step S3.4: Filter the overlapping area between the projection area and the worker's work area based on the distance-based edge exclusion rule, and finally determine whether the robotic arm constitutes a substantial intrusion by quantifying the overlapping area and using intelligent threshold judgment.

[0014] Preferably, step S3.3 includes:

[0015] in, Maximum height constraint; The vertical axis is the bottom coordinate of the robotic arm; The pixel area of ​​the robotic arm region; For regional adaptive mapping function, a multidimensional decision function based on normalized location and area ratio; , The height of the image frame; For a multidimensional decision function based on normalized location-area ratio, The total area of ​​the image. The area sensitivity coefficient, The perspective attenuation coefficient is... Let y be the center of the bottom of the robotic arm. The height of the image. These are the region mapping parameters; This is a minimum height constraint.

[0016] Preferably, step S4 includes: Step S4.1: Extract the center point from the obtained worker bounding box as the position feature, which represents the standing position of the person; Step S4.2: Based on the homogeneous coordinate transformation matrix M, the center point... Transform to the standard top-down plane coordinate system to obtain the ground position coordinates. ; Step S4.3: Based on the current ground position coordinates, compare them with the work area invaded by the robotic arm to determine whether the personnel are located in the current work area.

[0017] Preferably, step S4.3 includes:

[0018] Let be the number of intersections between any ray and the boundary of the worker's work area, where the ray is any ray emitted from the worker's position at an infinite distance; The number of rays; A function to determine whether a point is inside a polygon. The coordinates of the person's standing position on the ground. The NumPy array of the working area projected onto the robotic arm in the 2D image; when If the number of intersections is greater than 0, it indicates that personnel are within the work area; when When the number of intersections is 0, it indicates that the personnel are on the boundary of the work area; when If the number of intersections is less than 0, it means that the personnel are outside the work area.

[0019] A monitoring and real-time alarm system for the operational safety of a mobile robotic arm, provided by the present invention, includes: Module M1: Based on real-time video stream frames, it uses a target detection model to perform target detection and localization on the robotic arm and the worker; Module M2: Calculates the coordinates of the center point of the robotic arm based on its position coordinates; determines whether the robotic arm is in motion by comparing the pixel displacement of the center point of the robotic arm between consecutive video frames. Module M3: When the robotic arm is in motion, it determines whether the robotic arm is within the worker's work area based on the two-dimensional work area coordinate system; Module M4: Determines whether a worker is within the worker's work area based on a two-dimensional work area coordinate system.

[0020] Preferably, the module M3 includes: Module M3.1: Identifies and extracts the coordinates of the worker's work area on the ground from the current view of the camera; defines the two-dimensional coordinates of the current worker's work area on the real-world ground from a top-down perspective; calculates the homogeneous coordinate transformation matrix based on the coordinates of the worker's work area in the original image and the two-dimensional coordinates of the worker's work area on the real-world ground from a top-down perspective. The homogeneous coordinate transformation matrix is:

[0021] in, These are the original image coordinates. These are the transformed top-view plane coordinates, and s is a scaling factor; This is the homogeneous coordinate transformation matrix; Module M3.2: Based on the target region information of the robotic arm obtained by the target detection model, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Based on the homogeneous coordinate transformation matrix M, the point... Transform to the same two-dimensional coordinate system viewed from above to obtain its corresponding ground projection coordinates. This coordinate represents the horizontal center position of the robotic arm in the ground reference frame. Module M3.3: Obtaining the 2D ground projection point of the robotic arm Then, the projected height of the robotic arm is calculated, and the area of ​​the projected region is calculated based on the projected height of the robotic arm and the pixel area. Module M3.4: The edge exclusion rule based on distance filters the overlapping area between the projection area and the worker's work area, and determines whether the robotic arm constitutes a substantial intrusion by quantifying the overlapping area and using intelligent threshold judgment. Module M3.3 includes:

[0022] in, Maximum height constraint; The vertical axis is the bottom coordinate of the robotic arm; The pixel area of ​​the robotic arm region; For regional adaptive mapping function, a multidimensional decision function based on normalized location and area ratio; , The height of the image frame; For a multidimensional decision function based on normalized location-area ratio, The total area of ​​the image. The area sensitivity coefficient, The perspective attenuation coefficient is... Let y be the center of the bottom of the robotic arm. The height of the image. These are the region mapping parameters; This is a minimum height constraint.

[0023] Preferably, the module M4 includes: Module M4.1: Extracts the center point from the obtained worker bounding box as a location feature, which represents the worker's standing position; Module M4.2: Based on the homogeneous coordinate transformation matrix M, the center point... Transform to the standard top-down plane coordinate system to obtain the ground position coordinates. ; Module M4.3: Determines whether a person is within the current work area by comparing the current ground position coordinates with the work area intruded by the robotic arm; Module M4.3 includes:

[0024] Let be the number of intersections between any ray and the boundary of the worker's work area, where the ray is any ray emitted from the worker's position at an infinite distance; The number of rays; A function to determine whether a point is inside a polygon. The coordinates of the person's standing position on the ground. The NumPy array of the working area projected onto the robotic arm in the 2D image; when If the number of intersections is greater than 0, it indicates that personnel are within the work area; when When the number of intersections is 0, it indicates that the personnel are on the boundary of the work area; when If the number of intersections is less than 0, it means that the personnel are outside the work area.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention utilizes a low-cost monocular camera, combined with algorithms such as target detection, perspective change, height analysis, and edge exclusion, to achieve real-time two-dimensional accurate perception and dynamic behavior analysis of dangerous areas within a monitored scene. This significantly reduces security deployment costs and the burden of human supervision without requiring hardware modifications or relying on dense sensor deployments, effectively alleviating dependence on continuous manual supervision and improving the intelligence level and response efficiency of dangerous area management.

[0026] 2. This invention uses a single camera to capture video streams of the robotic arm's work area; utilizes a target detection model to identify the robotic arm and personnel in the video frames; continuously tracks the robotic arm's motion state through motion detection; applies perspective transformation technology to map the pixel coordinates of the image to the world coordinate system of the actual scene, accurately locating the worker's work area and the robotic arm's position, and can combine height determination for more accurate spatial analysis; finally, by calculating the spatial relationship between the personnel's position and the moving robotic arm in real time, a real-time alarm is immediately triggered when personnel intrude. Attached Figure Description

[0027] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 Flowchart of a method for monitoring and real-time alarming of mobile robotic arm operation safety.

[0028] Figure 2 This is a schematic diagram of the top-view projection coordinate system.

[0029] Figure 3 This is a schematic diagram for determining the three-dimensional spatial projection area of ​​a robotic arm based on its height.

[0030] Figure 4 A diagram illustrating how to determine whether a worker is in the work area. Detailed Implementation

[0031] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0032] Example 1 A monitoring and real-time alarm method for mobile robotic arm operation safety provided by the present invention includes: Step S1: Based on real-time video stream frames, use a target detection model to perform target detection and localization on the robotic arm and worker; Specifically, step S1 includes: Step S1.1: Using fixed cameras deployed at the work site, collect original images covering the entire working area of ​​the robotic arm and typical activity areas of the worker that meet preset requirements under different lighting conditions and time periods; perform preliminary screening of the collected original images, removing invalid images that do not contain the robotic arm or worker, and ensuring that images of the robotic arm in different postures and spatial positions, as well as images of the worker in different postures and in different areas, all account for a certain proportion in the dataset; after screening, finally construct a set of effective images containing a preset number; Step S1.2: Use the labeling tool, including LabelMe, to label each image in the obtained valid image set; for each robotic arm instance and each worker instance appearing in the image, use a rectangular bounding box to label the complete target area; when labeling, use the coordinates of the upper left and lower right corners of the target's bounding rectangle as the reference, and store the labeling results in the YOLO standard format. Each piece of information consists of the target category index and the normalized bounding box coordinates, thereby completing the construction of the dataset; Step S1.3: Build an object detection model based on the YOLOv8s architecture, train the object detection model based on the built dataset, and obtain the trained object detection model; Step S1.4: Based on real-time video stream frames, use the trained target detection model to perform target detection and localization on the robotic arm and the worker.

[0033] Step S2: Calculate the coordinates of the center point of the robotic arm based on its position coordinates; determine whether the robotic arm is in motion by comparing the pixel displacement of the center point of the robotic arm between consecutive video frames. Specifically, step S2 includes: when the displacement of the center point of the robotic arm position coordinates in two consecutive frames exceeds the pixel displacement threshold, the robotic arm is determined to be in a moving state;

[0034] Where t represents the number of frames. Let x be the center point of the robotic arm on the x-axis of frame t. Let y be the center point of the robotic arm on the t-th frame; Let x be the center point of the robotic arm on the x-axis of the (t-1)th frame; Let y be the center point of the robotic arm on the (t-1)th frame; if If the value exceeds the pixel displacement threshold, it is determined to be in a moving state; if If the value is less than or equal to the pixel displacement threshold, it is determined to be in a non-moving state.

[0035] Step S3: When the robotic arm is in motion, determine whether the robotic arm is within the worker's work area based on the two-dimensional work area coordinate system; Specifically, step S3 includes: Step S3.1: Identify and extract the coordinates of the worker's work area on the ground from the current view of the camera; define the two-dimensional coordinates of the current worker's work area on the real world ground from a top-down perspective; calculate the homogeneous coordinate transformation matrix based on the coordinates of the worker's work area in the original image and the two-dimensional coordinates of the worker's work area on the real world ground from a top-down perspective. The homogeneous coordinate transformation matrix is:

[0036] in, These are the original image coordinates. These are the transformed top-view plane coordinates, and s is a scaling factor; This is the homogeneous coordinate transformation matrix; Step S3.2: Based on the target region information of the robotic arm obtained by the target detection model, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Based on the homogeneous coordinate transformation matrix M, the point... Transform to the same two-dimensional coordinate system viewed from above to obtain its corresponding ground projection coordinates. This coordinate represents the horizontal center position of the robotic arm in the ground reference frame. Step S3.3: Obtain the two-dimensional ground projection point of the robotic arm. Then, the projected height of the robotic arm is calculated, and the area of ​​the projected region is calculated based on the projected height of the robotic arm and the pixel area.

[0037] in, Maximum height constraint; The vertical axis is the bottom coordinate of the robotic arm; The pixel area of ​​the robotic arm region; For regional adaptive mapping function, a multidimensional decision function based on normalized location and area ratio; , The height of the image frame; For a multidimensional decision function based on normalized location-area ratio, The total area of ​​the image. The area sensitivity coefficient, The perspective attenuation coefficient is... Let y be the center of the bottom of the robotic arm. The height of the image. These are the region mapping parameters; Minimum height constraint; Step S3.4: Filter the overlapping area between the projection area and the worker's work area based on the distance-based edge exclusion rule, and finally determine whether the robotic arm constitutes a substantial intrusion by quantifying the overlapping area and using intelligent threshold judgment.

[0038] Step S4: Determine whether the worker is within the worker's work area based on the two-dimensional work area coordinate system.

[0039] Specifically, step S4 includes: Step S4.1: Extract the center point from the obtained worker bounding box as the position feature, which represents the standing position of the person; Step S4.2: Based on the homogeneous coordinate transformation matrix M, the center point... Transform to the standard top-down plane coordinate system to obtain the ground position coordinates. ; Step S4.3: Based on the current ground position coordinates, compare them with the work area invaded by the robotic arm to determine whether the personnel are located in the current work area;

[0040] Let be the number of intersections between any ray and the boundary of the worker's work area, where the ray is any ray emitted from the worker's position at an infinite distance; The number of rays; A function to determine whether a point is inside a polygon. The coordinates of the person's standing position on the ground. The NumPy array of the working area projected onto the robotic arm in the 2D image; when If the number of intersections is greater than 0, it indicates that personnel are within the work area; when When the number of intersections is 0, it indicates that the personnel are on the boundary of the work area; when If the number of intersections is less than 0, it means that the personnel are outside the work area.

[0041] The present invention also provides a monitoring and real-time alarm system for the safety of mobile robotic arm operations. The monitoring and real-time alarm system for the safety of mobile robotic arm operations can be implemented by executing the process steps of the monitoring and real-time alarm method for the safety of mobile robotic arm operations. That is, those skilled in the art can understand the monitoring and real-time alarm method for the safety of mobile robotic arm operations as a preferred embodiment of the monitoring and real-time alarm system for the safety of mobile robotic arm operations.

[0042] Example 2 Example 2 is a preferred example of Example 1. The present invention provides a monitoring and real-time alarm method for the operational safety of mobile robotic arms, such as... Figures 1 to 4 As shown, it includes: Step 1: Detect and locate the robotic arm and worker targets based on the target detection model to accurately and in real time locate the spatial position of the robotic arm and worker in the visual scene; Specifically, step 1 includes: Step 1.1: Using fixed cameras deployed at the work site, collect raw images covering the entire working area of ​​the robotic arm and typical worker activity areas under different lighting conditions and time periods. Manually screen the collected raw images to remove invalid images that do not contain the robotic arm or worker, ensuring that images of the robotic arm in different poses and spatial positions, and of the worker in different poses and in different areas, constitute a certain proportion of the dataset to guarantee sample diversity and representativeness. After screening, a basic dataset containing a predetermined number of valid images is finally constructed.

[0043] Step 1.2: Use professional annotation tools such as LabelMe to perform fine-grained annotation on each image in the obtained basic dataset. Define two types of target labels: robotic arm and worker. For each instance of robotic arm and each instance of worker appearing in the image, use a bounding box to annotate, selecting the complete target region. During annotation, the coordinates of the upper left and lower right corners of the bounding rectangle of the target are used as the reference. Store the annotation results in the YOLO standard format. Each piece of information consists of the target category index class_index, corresponding to "robotic arm" or "worker", and the normalized bounding box coordinates (x1, y1, x2, y2), thus completing the preparation of the dataset.

[0044] Step 1.3: Select the YOLOv8s architecture as the base model to build an object detection model, and fine-tune the model by training with a dataset. Based on the characteristics of the robotic arm and worker targets, the input resolution of the model is uniformly set to 640×640 pixels. The training hyperparameters are configured as follows: the initial learning rate is set to 0.001; to improve data loading efficiency and fully utilize the parallel computing capabilities of the GPU, the number of data loading worker threads is set to 8; the total number of training epochs is set to 600. Through forward propagation, loss calculation, backpropagation, and parameter iterative updates, a performance-optimized object detection model weight file specifically for robotic arm and worker detection is finally obtained.

[0045] Step 1.4: In the application deployment phase, load the trained optimal robotic arm and worker target detection model. Preprocess and scale the real-time acquired or input video stream frames to 640×640 pixels before inputting them into the model. Set the detection confidence threshold to 0.5 and the non-maximum suppression (NMS) threshold to 0.5. The model infers from the input image, outputting bounding box information for all detected robotic arm and worker instances in the image. Each bounding box provides its top-left corner (x1, y1) and bottom-right corner (x2, y2) in pixel coordinates, thus achieving precise localization of the robotic arm and worker targets in the image.

[0046] Step 2: Determine if the robotic arm has moved; Specifically, step 2 includes: Based on the robotic arm's position coordinates output by the target detection model in step 1, calculate its center point coordinates. Determine if the robotic arm is in motion by comparing the pixel displacement of its center point between consecutive video frames. To avoid false positives due to slight jitter in the target detection bounding box, a pixel displacement threshold is set; the robotic arm is considered to be in motion only when the center point displacement exceeds this threshold for two consecutive frames. The pixel displacement formula is as follows:

[0047] Where t represents the number of frames. Let x be the center point of the robotic arm on the x-axis of frame t. Let y be the center point of the robotic arm on the t-th frame; The process branches based on the above judgment results. If If the value exceeds the pixel displacement threshold, it is determined to be a moving state, and step 3 is executed. If the value is less than the pixel displacement threshold, it is determined to be in a non-moving state. The current processing flow is then interrupted, and a predefined safe image is returned, ending the process.

[0048] Step 3: Determine if the robotic arm is within the worker's work area; Specifically, step 3 includes: Step 3.1: Establish a unified top-down projection coordinate system; identify and extract the worker's work area on the ground from the camera's current viewpoint. This area is usually a regular rectangle, for example: the area bounded by ground markers. Obtain the coordinates of its four corner points in the image pixel coordinate system, denoted as . Subsequently, a two-dimensional coordinate system is defined for the work area on the real-world ground, for example, with a corner of the area as the origin and the length and width directions as the X and Y axes. Under this coordinate system, the work area is normalized into an axis-aligned rectangle, and the coordinates of its four corner points can be defined as follows: Where W and L are the actual width and length of the work area, respectively. Using these two sets of corresponding points, a 3x3 homogeneous coordinate transformation matrix M is calculated through perspective transformation and stored as a NumPy array to complete the establishment of the work area's reference plane. The transformation matrix formula is as follows:

[0049] in, These are the original image coordinates. is the transformed top-view plane coordinate, and s is a scaling factor.

[0050] Based on the target area information of the robotic arm returned in step 1, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Applying the perspective transformation matrix M obtained above, the points are... Transform to the same top-down 2D coordinate system to obtain its corresponding ground projection coordinates. These coordinates represent the horizontal center position of the robotic arm in the ground reference frame.

[0051] Step 3.2: Obtain the two-dimensional ground projection point of the robotic arm Next, it is necessary to further determine whether the robotic arm structure itself intrudes into the worker's work area in three-dimensional space, that is, to consider the projected area caused by the height. This step introduces the consideration of the height dimension, and the core formula for height calculation is as follows:

[0052] in: Maximum height constraint; : Vertical axis of the bottom of the robotic arm; : The pixel area of ​​the robotic arm region, i.e. ; : Region adaptive mapping function, a multidimensional decision function based on normalized location-area ratio, used to improve spatial adaptability. , The height of the image frame.

[0053] Through a dual-constraint guarantee mechanism, namely upper limit constraint To prevent calculation results from exceeding the physical limits and lower bounds Ensure that each area meets the minimum safety height requirements; more specifically, Upper limit constraint: ） The upper limit constraint needs to consider three aspects: Area counter-compensation item: That is, the larger the area of ​​the robotic arm region ( The larger the value, the closer the target is to the camera, and the less height compensation is required. It is an area sensitivity coefficient, used to balance the area; Perspective correction items: The lower the position ( The larger the value, the more significant the perspective effect, and the height value decreases accordingly. β is the perspective attenuation coefficient. Region scaling option: Non-linear adjustments are made based on the characteristics of the scene area to improve spatial adaptability. These are the parameters for region mapping.

[0054] Lower bound constraint: Based on the principle that objects appear larger when closer and smaller when farther away, ensure that each area meets the minimum safety height requirements.

[0055] Step 3.3: Precise Region Determination Based on Edge Exclusion Rules: The preliminary determination in Step 3.2 may result in false edge detection due to the intersection of the projected area's shape (circular) with the rectangular work area's edge. That is, even if the actual robotic arm body has not fully entered the work area, an alarm may be triggered simply because the circular boundary of its projected area slightly overlaps with the rectangular boundary, potentially leading to a higher false alarm rate. To address this issue, this step introduces a distance-based edge exclusion rule for precise filtering. By quantifying the overlapping area and applying intelligent threshold judgment, it ultimately determines whether the robotic arm constitutes a substantial intrusion and obtains the corresponding NumPy array of the work area. .

[0056] The process branches based on the above judgment results. If it is determined that the robotic arm is in the working area, then proceed to step 4. If it is determined that the robotic arm is in the non-working area, then the current processing flow is interrupted, a predefined safety image is returned, and the process ends.

[0057] Step 4: Determine if the worker is in the work area; Specifically, step 4 includes: This step, based on the detection results of step 1 and the coordinate system established in step 3, performs a compliance assessment of the workers. The bottom center point is extracted as a location feature from the worker bounding box obtained in step 1. This point represents the position where the person is standing. Applying the perspective transformation matrix M established in step 3.1, the point... Transform to the standard top-down plane coordinate system to obtain accurate ground position coordinates. The ground coordinates are compared with the work area identified as intruded upon by the robotic arm in step 3. A point-polygon inclusion detection algorithm from computational geometry is used to determine whether a person is located within this work area. The formula is as follows:

[0058] The process branches based on the above judgment results. If the worker is determined to be in the work area, a predefined alarm image is returned and an alarm signal is issued. If the worker is determined to be in a non-work area, a predefined safety image is returned, and the process ends.

[0059] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0060] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A monitoring and real-time alarming method for mobile robot arm operation safety, characterized in that, include: Step S1: Based on real-time video stream frames, use a target detection model to perform target detection and localization on the robotic arm and worker; Step S2: Calculate the coordinates of the center point of the robotic arm based on its position coordinates; determine whether the robotic arm is in motion by comparing the pixel displacement of the center point of the robotic arm between consecutive video frames. Step S3: When the robotic arm is in motion, determine whether the robotic arm is within the worker's work area based on the two-dimensional work area coordinate system; Step S4: Determine whether the worker is within the worker's work area based on the two-dimensional work area coordinate system.

2. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 1, characterized in that, Step S1 includes: Step S1.1: Using fixed cameras deployed at the work site, collect original images covering the entire working area of ​​the robotic arm and typical activity areas of the worker that meet preset requirements under different lighting conditions and time periods; perform preliminary screening of the collected original images, removing invalid images that do not contain the robotic arm or worker, and ensuring that images of the robotic arm in different postures and spatial positions, as well as images of the worker in different postures and in different areas, all account for a certain proportion in the dataset; after screening, finally construct a set of effective images containing a preset number; Step S1.2: Use the labeling tool, including LabelMe, to label each image in the obtained valid image set; for each robotic arm instance and each worker instance appearing in the image, use a rectangular bounding box to label the complete target area; when labeling, use the coordinates of the upper left and lower right corners of the target's bounding rectangle as the reference, and store the labeling results in the YOLO standard format. Each piece of information consists of the target category index and the normalized bounding box coordinates, thereby completing the construction of the dataset; Step S1.3: Build an object detection model based on the YOLOv8s architecture, train the object detection model based on the built dataset, and obtain the trained object detection model; Step S1.4: Based on real-time video stream frames, use the trained target detection model to perform target detection and localization on the robotic arm and the worker.

3. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 1, characterized in that, Step S2 includes: when the displacement of the center point of the robotic arm position coordinates in two consecutive frames exceeds the pixel displacement threshold, the robotic arm is determined to be in a moving state. Where t represents the number of frames. Let x be the center point of the robotic arm on the x-axis of frame t. Let y be the center point of the robotic arm on the t-th frame; Let x be the center point of the robotic arm on the x-axis of the (t-1)th frame; Let y be the center point of the robotic arm on the (t-1)th frame; if If the value exceeds the pixel displacement threshold, it is determined to be in a moving state; if If the value is less than or equal to the pixel displacement threshold, it is determined to be in a non-moving state.

4. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 1, characterized in that, Step S3 includes: Step S3.1: Identify and extract the coordinates of the worker's work area on the ground from the current view of the camera; define the two-dimensional coordinates of the current worker's work area on the real world ground from a top-down perspective; calculate the homogeneous coordinate transformation matrix based on the coordinates of the worker's work area in the original image and the two-dimensional coordinates of the worker's work area on the real world ground from a top-down perspective. The homogeneous coordinate transformation matrix is: in, These are the original image coordinates. These are the transformed top-view plane coordinates, and s is a scaling factor; This is the homogeneous coordinate transformation matrix; Step S3.2: Based on the target region information of the robotic arm obtained by the target detection model, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Based on the homogeneous coordinate transformation matrix M, the point... Transform to the same two-dimensional coordinate system viewed from above to obtain its corresponding ground projection coordinates. This coordinate represents the horizontal center position of the robotic arm in the ground reference frame. Step S3.3: Obtain the two-dimensional ground projection point of the robotic arm. Then, the projected height of the robotic arm is calculated, and the area of ​​the projected region is calculated based on the projected height of the robotic arm and the pixel area. Step S3.4: Filter the overlapping area between the projection area and the worker's work area based on the distance-based edge exclusion rule, and finally determine whether the robotic arm constitutes a substantial intrusion by quantifying the overlapping area and using intelligent threshold judgment.

5. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 4, characterized in that, Step S3.3 includes: in, Maximum height constraint; The vertical axis is the bottom coordinate of the robotic arm; The pixel area of ​​the robotic arm region; For regional adaptive mapping function, a multidimensional decision function based on normalized location and area ratio; , The height of the image frame; For a multidimensional decision function based on normalized location-area ratio, The total area of ​​the image. The area sensitivity coefficient, The perspective attenuation coefficient is... Let y be the center of the bottom of the robotic arm. The height of the image. These are the region mapping parameters; This is a minimum height constraint.

6. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 1, characterized in that, Step S4 includes: Step S4.1: Extract the center point from the obtained worker bounding box as the position feature, which represents the standing position of the person; Step S4.2: Based on the homogeneous coordinate transformation matrix M, the center point... Transform to the standard top-down plane coordinate system to obtain the ground position coordinates. ; Step S4.3: Based on the current ground position coordinates, compare them with the work area invaded by the robotic arm to determine whether the personnel are located in the current work area.

7. The monitoring and real-time alarm method for mobile robotic arm operation safety according to claim 6, characterized in that, Step S4.3 includes: Let be the number of intersections between any ray and the boundary of the worker's work area, where the ray is any ray emitted from the worker's position at an infinite distance; The number of rays; A function to determine whether a point is inside a polygon. The coordinates of the person's standing position on the ground. The NumPy array of the working area projected onto the robotic arm in the 2D image; when If the number of intersections is greater than 0, it indicates that personnel are within the work area; when When the number of intersections is 0, it indicates that the personnel are on the boundary of the work area; when If the number of intersections is less than 0, it means that the personnel are outside the work area.

8. A monitoring and real-time alarm system for the operational safety of mobile robotic arms, characterized in that, include: Module M1: Based on real-time video stream frames, it uses a target detection model to perform target detection and localization on the robotic arm and the worker; Module M2: Calculates the coordinates of the center point of the robotic arm based on its position coordinates; determines whether the robotic arm is in motion by comparing the pixel displacement of the center point of the robotic arm between consecutive video frames. Module M3: When the robotic arm is in motion, it determines whether the robotic arm is within the worker's work area based on the two-dimensional work area coordinate system; Module M4: Determines whether a worker is within the worker's work area based on a two-dimensional work area coordinate system.

9. The monitoring and real-time alarm system for mobile robotic arm operation safety according to claim 8, characterized in that, The module M3 includes: Module M3.1: Identifies and extracts the coordinates of the worker's work area on the ground from the current view of the camera; defines the two-dimensional coordinates of the current worker's work area on the real-world ground from a top-down perspective; calculates the homogeneous coordinate transformation matrix based on the coordinates of the worker's work area in the original image and the two-dimensional coordinates of the worker's work area on the real-world ground from a top-down perspective. The homogeneous coordinate transformation matrix is: in, These are the original image coordinates. These are the transformed top-view plane coordinates, and s is a scaling factor; This is the homogeneous coordinate transformation matrix; Module M3.2: Based on the target region information of the robotic arm obtained by the target detection model, obtain the pixel coordinates of the center point of the bottom of the robotic arm in the current camera image. Based on the homogeneous coordinate transformation matrix M, the point... Transform to the same two-dimensional coordinate system viewed from above to obtain its corresponding ground projection coordinates. This coordinate represents the horizontal center position of the robotic arm in the ground reference frame. Module M3.3: Obtaining the 2D ground projection point of the robotic arm Then, the projected height of the robotic arm is calculated, and the area of ​​the projected region is calculated based on the projected height of the robotic arm and the pixel area. Module M3.4: The edge exclusion rule based on distance filters the overlapping area between the projection area and the worker's work area, and determines whether the robotic arm constitutes a substantial intrusion by quantifying the overlapping area and using intelligent threshold judgment. Module M3.3 includes: in, Maximum height constraint; The vertical axis is the bottom coordinate of the robotic arm; The pixel area of ​​the robotic arm region; For regional adaptive mapping function, a multidimensional decision function based on normalized location and area ratio; , The height of the image frame; For a multidimensional decision function based on normalized location-area ratio, The total area of ​​the image. The area sensitivity coefficient, The perspective attenuation coefficient is... Let y be the center of the bottom of the robotic arm. The height of the image. These are the region mapping parameters; This is a minimum height constraint.

10. The monitoring and real-time alarm system for mobile robotic arm operation safety according to claim 8, characterized in that, The module M4 includes: Module M4.1: Extracts the center point from the obtained worker bounding box as a location feature, which represents the worker's standing position; Module M4.2: Based on the homogeneous coordinate transformation matrix M, the center point... Transform to the standard top-down plane coordinate system to obtain the ground position coordinates. ; Module M4.3: Determines whether a person is within the current work area by comparing the current ground position coordinates with the work area intruded by the robotic arm; Module M4.3 includes: Let be the number of intersections between any ray and the boundary of the worker's work area, where the ray is any ray emitted from the worker's position at an infinite distance; The number of rays; A function to determine whether a point is inside a polygon. The coordinates of the person's standing position on the ground. The NumPy array of the working area projected onto the robotic arm in the 2D image; when If the number of intersections is greater than 0, it indicates that personnel are within the work area; when When the number of intersections is 0, it indicates that the personnel are on the boundary of the work area; when If the number of intersections is less than 0, it means that the personnel are outside the work area.

Images