Industrial machinery operation safety monitoring method and system, storage medium and electronic equipment

Abstract

The invention discloses a monitoring method and system for operation safety of an industrial machine, a storage medium and electronic equipment. The method comprises the following steps: acquiring and displaying a three-dimensional alarm area of the industrial machine in an imaging visual field of image data acquisition equipment; acquiring and displaying position information of a human body target appearing in the imaging view; and alarm processing is carried out after the alarm signal is received. According to the invention, the alarm accuracy of industrial machinery operation safety monitoring can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of industrial safety technology, and in particular to monitoring methods, devices, storage media and electronic equipment for the safety of industrial machinery operations. Background Technology

[0002] In modern industrial production, mechanical devices such as collaborative arms, machine tools, and automated equipment are widely used in manufacturing, assembly lines, warehouse management, and energy production. These devices can perform precise, repetitive tasks and improve production efficiency, but because operators need to work nearby, there are potential safety risks. According to ISO / TS 15066 and ISO 10218-1 / 2 standards, the warning zone for mechanical devices is typically defined as the maximum range of motion of the device plus a buffer zone (e.g., 0.5-1 meter) to prevent operators from colliding with the device.

[0003] In existing technologies, security monitoring mainly relies on devices such as laser scanners, infrared sensors, and pressure pads. While these methods can achieve basic intrusion detection, they have significant limitations: laser scanners have limited coverage and are expensive; infrared sensors are susceptible to environmental interference, such as temperature fluctuations or dust; and pressure pads are only suitable for ground detection and cannot handle three-dimensional space. Furthermore, traditional systems often require coordinate system calibration of the camera and mechanical devices, which increases deployment complexity and maintenance costs, especially in scenarios where device positions change dynamically, making calibration prone to failure. Moreover, existing surveillance cameras struggle to maintain high robustness and real-time response, exhibiting high false alarm and false negative rates, failing to meet the zero-tolerance requirements of industrial security. Summary of the Invention

[0004] This application provides a method, device, storage medium, and electronic equipment for monitoring the safety of industrial machinery operations, which can effectively improve the alarm accuracy of industrial machinery operation safety monitoring.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] A method for monitoring the safety of industrial machinery operation includes:

[0007] Acquire and display the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device;

[0008] Acquire and display the location information of human targets appearing within the imaging field of view;

[0009] Upon receiving an alarm signal, an alarm will be processed.

[0010] The three-dimensional alarm zone is determined based on the ground alarm zone configuration of the industrial machinery and the configuration of the fence posts. The position of the human target is determined by target detection of the human target in the three-dimensional point cloud image of the work site in the current frame. The three-dimensional point cloud image is a combination of two-dimensional image and depth information determined by the image data acquisition device at the work site. The alarm signal is generated after determining the positioning result of the corresponding human target inside and outside the fence based on the three-dimensional position information of the human target and the three-dimensional alarm zone, and the alarm is triggered based on the positioning result.

[0011] Preferably, the method for determining the positioning result of the corresponding human target inside or outside the fence includes: determining the effective three-dimensional point of the human target based on the three-dimensional position information, and determining the positioning result of the corresponding human target inside or outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area.

[0012] Preferably, the method for obtaining the three-dimensional position information includes: using a pre-trained human body tracking model to perform target detection and tracking on the three-dimensional point cloud image, and outputting a corresponding segmentation mask and trajectory information for each human body target detected in the three-dimensional point cloud image;

[0013] The determination of the valid 3D points is based on the trajectory information and the segmentation mask.

[0014] Preferably, the human body tracking model includes a multi-task detection sub-model and a tracking sub-model;

[0015] In the multi-task detection sub-model, target detection processing is performed on the 3D point cloud image. A segmentation mask is determined for each detected human target, and its corresponding human target bounding box and at least one specified part target bounding box are predicted. Based on the human target bounding box and at least one specified part target bounding box, REID features for personnel tracking are determined.

[0016] In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

[0017] Preferably, in the multi-task detection sub-model, the target category is predicted through an attention mechanism, and a corresponding convolutional kernel is generated for each target; the first target is segmented based on the convolutional kernel corresponding to the first target classified as human body, and a segmentation mask corresponding to the first target is obtained; based on the convolutional kernel corresponding to the first target, the human body target box and at least one specified part target box corresponding to the first target are predicted by predicting the target box corner heatmap; the segmentation mask corresponding to the first target is multiplied with the image features of the current frame image, and post-processed to obtain the REID features of the first target.

[0018] Preferably, the similarity calculation method between the first target and the first historical trajectory includes:

[0019] Using the features of the first target as the center, calculate the first matching probability between the first target and all historical trajectories;

[0020] Using the features of the first trajectory as the center, calculate the second matching probability between the first trajectory and each human target in the current frame;

[0021] The average of the first matching probability and the second matching probability is used as the similarity between the first target and the first historical trajectory.

[0022] Preferably, the prediction of obtaining the human body bounding box and at least one specified body part bounding box corresponding to the first target includes:

[0023] Based on the convolution kernel corresponding to the first target, the probability of each target point being a corner point of any target box is predicted. The probability is used as the weight of the corresponding target point, and the weighted average of the coordinates of each target point is calculated. The calculation result is used as the corner point of any target box. The any target box is one of the human body target box corresponding to the first target and at least one target box of a specified part.

[0024] Preferably, after performing target detection on the three-dimensional point cloud image, the human target bounding box corresponding to each human target is further determined;

[0025] After determining that the first human target is located within the fence or within a set distance from the fence in the current frame, and before determining to trigger an alarm based on the location result, the method further includes:

[0026] The human target bounding box corresponding to the first human target and the image in which it is located are input into the pre-trained false detection removal model; in the false detection removal model, feature extraction is performed based on the input image and the bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection base library to determine whether the current frame of the first human target is a false detection target and its confidence level.

[0027] When determining whether an alarm is triggered, further analysis is performed based on the first result and its confidence level.

[0028] The false detection base database stores the registered false detection target information, including: the target bounding box information of the registered false detection target input by the user, and the embedding vector of the registered false detection target obtained by processing the target bounding box of the registered false detection target and the image where the target bounding box is located using the false detection model.

[0029] Preferably, after performing target detection on the three-dimensional point cloud image, at least one designated part of the target box corresponding to each human body target is further determined;

[0030] In the false detection removal model, feature extraction is further performed based on the at least one specified part target box, and binary classification is performed based on the feature extraction results to predict whether the first human target in the current frame is indeed a human target; when determining to trigger an alarm, it is further based on the prediction results and their confidence levels.

[0031] Preferably, the false detection removal model includes a backbone network for extracting image features, a first branch network for binary classification based on the extracted image features, and a second branch network for retrieval based on the extracted image features and the registered false detection target information;

[0032] When training the false detection removal model, the first branch network performs binary classification on the extracted image features and uses a first loss function to minimize the difference between the predicted probability distribution and the true distribution; the second branch network maps the extracted image features to an embedding space and uses a second loss function to optimize the embedding space, so that similar targets are clustered and dissimilar targets are separated.

[0033] The training data for the false detection removal model includes two groups of training data. The first group of training data is processed using the backbone network and the first branch network to calculate the value of the first loss function. The second group of training data is processed using the backbone network and the second branch network to calculate the value of the second loss function. The weighted sum of the values ​​of the first and second loss functions is then used as the value of the joint loss function to update the model parameters.

[0034] Preferably, the method of determining the three-dimensional alarm zone of the industrial machinery based on the ground alarm zone configuration and the fence post configuration includes:

[0035] Based on the ground alarm zone configuration, a regular polygonal region of the ground in the world coordinate system is determined.

[0036] Based on the configuration of the fence posts, determine the projection point of the straight line where the fence posts are located to the ground, and update the vertices of the polygonal regular area on the ground based on the projection point;

[0037] A ground plane is obtained by fitting the vertices of the regular polygonal region of the ground, and all the vertices are projected onto the ground plane to generate the edges of the ground plane.

[0038] The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

[0039] Preferably, before acquiring and displaying the three-dimensional alarm area, the method further includes: receiving the ground alarm area configuration and / or the fence post configuration input by the user through a configuration interface;

[0040] or,

[0041] The configuration of the ground alarm zone and / or the configuration of the fence posts are determined based on the data collected by the image data acquisition device.

[0042] Preferably, when generating the side of the fence, the three-dimensional points of the fence posts are retained when the straight line where the fence posts are located is determined based on the configuration of the fence posts; if the fitted fence post is located outside the three-dimensional alarm area, the fence post is ignored.

[0043] Preferably, determining the positioning result of the corresponding human target inside or outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area includes:

[0044] The relative positional relationship between each effective three-dimensional point and the side of the three-dimensional alarm area is determined point by point, and the positioning result of the corresponding human target inside and outside the fence is determined based on the judgment results of all effective three-dimensional points.

[0045] Preferably, determining to trigger an alarm based on the positioning result includes:

[0046] Based on the positioning results corresponding to multiple frames and the corresponding confidence scores, the confidence scores of the human target bounding box and at least one specified part target bounding box, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information, it is determined whether to trigger an alarm.

[0047] Preferably, the method further includes: if there is a closed blind zone at the work site, acquiring and displaying the number of human targets located in the closed blind zone; and when the count result is not 0, performing an alarm process.

[0048] A monitoring system for the safety of industrial machinery operation includes: an image data acquisition device, a processor, and a display;

[0049] The image data acquisition device is used to collect data at the work site;

[0050] The processor is configured to determine the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device based on the ground alarm area configuration and fence post configuration of the industrial machinery; it is also configured to acquire a three-dimensional point cloud image of the work site in the current frame, and acquire the three-dimensional position information of the human target detected in the three-dimensional point cloud image in the current frame; it is further configured to determine the positioning result of the corresponding human target inside and outside the fence in the current frame based on the three-dimensional position information of the human target and the three-dimensional alarm area, and determine whether to trigger an alarm based on the positioning result, and notify the display when it is determined that an alarm is triggered; wherein, the three-dimensional point cloud image is a combination of two-dimensional image and depth information determined based on the data;

[0051] The display is used to acquire and display the three-dimensional alarm area of ​​the industrial machinery, and also to acquire and display the position information of human targets appearing within the imaging field of view, and to perform alarm processing after receiving an alarm signal.

[0052] Preferably, in the processor, determining the positioning result of the corresponding human target inside or outside the fence includes: determining the effective three-dimensional point of the human target based on the three-dimensional position information, and determining the positioning result of the corresponding human target being inside or outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area.

[0053] Preferably, in the processor, the method of acquiring the three-dimensional position information includes: using a pre-trained human body tracking model to perform target detection and tracking on the three-dimensional point cloud image, and outputting a corresponding segmentation mask and trajectory information for each human body target detected in the three-dimensional point cloud image;

[0054] In the positioning unit, the effective three-dimensional points are determined based on the trajectory information and the segmentation mask.

[0055] Preferably, in the processor, the human body tracking model includes a multi-task detection sub-model and a tracking sub-model;

[0056] In the multi-task detection sub-model, target detection processing is performed on the 3D point cloud image. A segmentation mask is determined for each detected human target, and its corresponding human target bounding box and at least one specified part target bounding box are predicted. Based on the human target bounding box and at least one specified part target bounding box, REID features for personnel tracking are determined.

[0057] In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

[0058] Preferably, in the processor, after performing target detection on the 3D point cloud image, the processor further determines the human bounding box corresponding to each human target; it also inputs the human bounding box corresponding to the first human target and its image into a pre-trained false detection removal model after determining that the current frame of the first human target is located within the fence or within a set distance from the fence; in the false detection removal model, feature extraction is performed based on the input image and bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection base library to determine whether the current frame of the first human target is a false detection target and its confidence level; further, based on the first result and its confidence level, it determines whether to trigger an alarm.

[0059] The false detection base database stores the registered false detection target information, including: the target bounding box information of the registered false detection target input by the user, and the embedding vector of the registered false detection target obtained by processing the target bounding box of the registered false detection target and the image where the target bounding box is located using the false detection model.

[0060] Preferably, in the processor, after performing target detection on the 3D point cloud image, it is further used to determine at least one designated part bounding box corresponding to each human target; it is also used in the false detection removal model to further perform feature extraction based on the at least one designated part bounding box, and to perform binary classification processing based on the feature extraction result to predict whether the first human target in the current frame is indeed a human target; and to further determine whether to trigger an alarm based on the prediction result and its confidence level.

[0061] Preferably, in the processor, determining the three-dimensional alarm zone of the industrial machinery based on the ground alarm zone configuration and the fence post configuration includes:

[0062] Based on the ground alarm zone configuration, a regular polygonal region of the ground in the world coordinate system is determined.

[0063] Based on the configuration of the fence posts, determine the projection point of the straight line where the fence posts are located to the ground, and update the vertices of the polygonal regular area on the ground based on the projection point;

[0064] A ground plane is obtained by fitting the vertices of the regular polygonal region of the ground, and all the vertices are projected onto the ground plane to generate the edges of the ground plane.

[0065] The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

[0066] Preferably, the display is further configured to receive the ground alarm zone configuration and / or the fence post configuration input by the user through the configuration interface, and forward it to the processor;

[0067] or,

[0068] The processor is further configured to determine the ground alarm zone configuration and / or the fence post configuration based on the data acquired by the image data acquisition device.

[0069] Preferably, the processor is further configured to retain, when generating the fence side, the three-dimensional points of the fitted posts when determining the straight line where the fence posts are located based on the fence post configuration; if the fitted fence post is located outside the three-dimensional alarm area, then the fence post is ignored.

[0070] Preferably, in the processor, determining the positioning result of the corresponding human target inside and outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area includes:

[0071] The relative positional relationship between each effective three-dimensional point and the side of the three-dimensional alarm area is determined point by point, and the relative positional relationship between the corresponding human target and the three-dimensional alarm area is determined based on the determination results of all effective three-dimensional points.

[0072] Preferably, in the processor, determining whether to trigger an alarm based on the positioning result includes:

[0073] Based on the positioning results corresponding to multiple frames and the corresponding confidence scores, the confidence scores of the human target bounding box and at least one specified part target bounding box, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information, it is determined whether to trigger an alarm.

[0074] Preferably, when there is a closed blind zone at the work site, the processor is further configured to count human targets located within the closed blind zone and send the counting result to the display; when the counting result is not 0, an alarm is triggered and the display is notified.

[0075] The display is used to acquire and display the number of human targets located in the closed blind area when there is a closed blind area at the work site; and to perform an alarm process when it receives a notification that the result of the processor is not 0.

[0076] A computer-readable storage medium storing computer instructions thereon, characterized in that, when the instructions are executed by a processor, they can implement the monitoring method for industrial machinery operation safety described in any of the preceding claims.

[0077] An electronic device, characterized in that the electronic device includes at least a computer-readable storage medium and a processor;

[0078] The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the monitoring method for industrial machinery operation safety as described above.

[0079] As can be seen from the above technical solution, in this application, the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device is acquired and displayed; the position information of the human target appearing within the imaging field of view of the image data acquisition device is acquired and displayed, and alarm processing is performed after receiving an alarm signal; wherein, the three-dimensional alarm area is determined based on the ground alarm area configuration and fence post configuration of the industrial machinery, thereby effectively defining the lateral position of the alarm area; the position information of the human target is the three-dimensional position information of the human target determined by target detection of the three-dimensional point cloud image of the work site in the current frame; the alarm signal is generated after determining the positioning result of the corresponding human target inside and outside the fence in the current frame based on the three-dimensional position information of the human target and the three-dimensional alarm area, and determining the alarm trigger based on the positioning result; thus, through the processing of this application, on the one hand, a three-dimensional alarm area is established, and on the other hand, the three-dimensional position of the human target in space is acquired using the image data acquisition device, thereby positioning is performed through the relative positional relationship between the human target and the three-dimensional alarm area, thereby effectively improving the positioning accuracy, and alarm judgment is made based on the more accurate positioning result, thereby effectively improving the alarm accuracy of industrial machinery operation safety monitoring. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the basic method for monitoring the safety of industrial machinery operations in this application;

[0081] Figure 2 This is a schematic diagram illustrating the specific process of the monitoring method for industrial machinery operation safety in a particular embodiment of this application;

[0082] Figure 3 This is a schematic diagram of the system modules of the security monitoring method in a specific embodiment;

[0083] Figure 4 Example diagram of ground alarm zone configuration;

[0084] Figure 5 This is a schematic diagram of the structure of the multi-task detection sub-model in the embodiments of this application;

[0085] Figure 6 This is a schematic diagram of REID feature extraction.

[0086] Figure 7 This is a schematic diagram of the basic structure of the monitoring device in this application;

[0087] Figure 8 This is a schematic diagram of the basic structure of the electronic device in this application. Detailed Implementation

[0088] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0089] In existing methods for security monitoring using stereo vision, the location of a detected target within or outside the alarm zone is determined based on the absolute positioning of the stereo vision. However, the absolute positioning error of stereo vision can affect the accuracy of the positioning results, and consequently, the accuracy of the alarm. Specifically, on the one hand, stereo vision positioning itself has errors. When the object is far from the camera, the parallax is small, so the depth error is relatively large. This is because the smaller the parallax, the worse the accuracy of depth calculation, leading to an increase in ranging error. On the other hand, camera height, pitch angle, and tilt angle errors also affect positioning. Among them, pitch angle error leads to longitudinal positioning deviation and height error, which is approximately proportional to the target distance. Tilt angle error leads to lateral positioning deviation and height error, which is also approximately proportional to the target distance. Furthermore, the absolute positioning error of human targets also affects positioning accuracy. The absolute positioning error of human targets is related to the human's posture and orientation. The human point cloud is the visible part of the human surface points, not the complete human point cloud. If the actual distance between the target and the fence is less than 10cm, there will be no gap between the target and the fence in the positioning result. For targets at height (e.g., a person standing on a 1m high platform near the fence boundary), the error of projecting onto the ground alarm area is large due to the distance from the ground. Moreover, the ground alarm area itself may have errors, especially when the intersection line between the fence and the ground is obscured. The ground alarm area is estimated, which has low accuracy.

[0090] By analyzing the above errors, this application expands the existing ground alarm area into a three-dimensional alarm area and obtains the three-dimensional position information of the human target. By comparing the relative positional relationship between the human target and the three-dimensional alarm area, the positioning relationship between the human target and the fence around the machinery is determined, thus avoiding the cumulative error caused by positioning through absolute positional relationship and the cumulative error of coordinate system calibration.

[0091] Figure 1 This is a flowchart illustrating the basic method for monitoring the safety of industrial machinery operations in this application. Figure 1 As shown, the method includes:

[0092] Step 101: Acquire and display the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device.

[0093] The three-dimensional alarm zone of industrial machinery can be determined based on the configuration of the ground alarm zone of the industrial machinery and the configuration of the fence posts of the industrial machinery.

[0094] Existing safety monitoring methods rely solely on ground-based alarm zones to determine whether a human target is within the alarm area. This application, however, extends the alarm zone to three-dimensional space, defining a three-dimensional alarm zone for the operating range of industrial machinery. A stereoscopic vision camera acquires the target's three-dimensional spatial position, and then the target's three-dimensional spatial position is compared relative to the three-dimensional alarm zone to improve the accuracy of determining whether the target is within the alarm zone.

[0095] Specifically, this application extends the alarm area to three-dimensional space based on the fence post configuration for industrial machinery. Here, "fence" refers to a physical fence surrounding the industrial machinery. More specifically, the positions of the fence posts can be determined based on the fence post configuration and combined with the ground alarm area to define the three-dimensional alarm area. The fence post configuration can be configurable parameters input by the user through a configuration interface, or it can be automatically generated based on on-site data collected by image data acquisition equipment and processed using a neural network.

[0096] Image data acquisition equipment is used to collect images and data from the work site to determine 3D point cloud images. Image acquisition equipment can be implemented in various ways, such as stereo vision cameras, or radar cameras (including radar and cameras), etc.

[0097] This application will display the identified 3D alarm area.

[0098] Step 102: Acquire and display the location information of the human target appearing in the imaging field of view of the image data acquisition device.

[0099] Among them, the human target location information is the three-dimensional location information of the human target determined by target detection in the three-dimensional point cloud image of the work site in the current frame. The three-dimensional point cloud image is a combination of two-dimensional image and depth information determined by the image data acquisition equipment at the work site.

[0100] Specifically, the image data acquisition device collects data from the work site. A 3D point cloud image can be generated based on this data, which may include 2D image data (e.g., RGB format) and depth information of each 2D coordinate point. Alternatively, the 3D point cloud image can be determined within the image data acquisition device and then sent to the processor; or, the image data acquisition device can send the acquired data to the processor, which then determines the 3D point cloud image based on the acquired data.

[0101] The processor detects human targets and their 3D position information in a 3D point cloud image. The processing of detecting the human target and its 3D position information can be implemented using existing methods, such as target detection neural networks.

[0102] The processing of data acquired by image acquisition devices can be done in units of image frames.

[0103] Step 103: After receiving the alarm signal, perform alarm processing.

[0104] The alarm signal is generated based on the three-dimensional position information of the human target and the location result of the corresponding human target inside and outside the fence, which is determined by the three-dimensional alarm area, and the alarm signal is generated after determining the location result that needs to be triggered.

[0105] Specifically, based on the aforementioned three-dimensional location information and three-dimensional alarm area, the positioning result of the corresponding human target inside and outside the fence is determined, and then based on the positioning result, it is determined whether an alarm needs to be triggered. When it is determined that an alarm needs to be triggered, an alarm signal is generated.

[0106] In the acquired three-dimensional position information of human targets, considering the different amounts of data to be processed and the reliability of the data, it is optional that after detecting the three-dimensional position information of each human target in the current frame, the effective three-dimensional points of the corresponding human targets in the current frame can be determined based on the three-dimensional position information, that is, the position data of human target positioning; then, based on the relative positional relationship between the effective three-dimensional points and the three-dimensional alarm area, the positioning result of the human target inside and outside the fence is determined.

[0107] The alarm is triggered based on the given location results. In its simplest form, an alarm can be triggered if the location is determined to be inside the fence, and not triggered if the location is determined to be outside the fence.

[0108] Alternatively, when the positioning results determine that a human target is near a fence, for example, within a set range from the fence, further analysis can be performed based on the confidence level of the human target's three-dimensional position information in the current frame and the confidence level of the positioning results, and compared with the set alarm strategy to determine whether to trigger an alarm.

[0109] Thus far, in this application Figure 1 The basic process of the safety monitoring method shown is now complete. As described above, through the processing of this application, on the one hand, the three-dimensional alarm area is acquired and displayed, and on the other hand, the three-dimensional position of the human target in space is acquired and displayed. Positioning is performed by the relative positional relationship between the human target and the three-dimensional alarm area, which effectively reduces the positioning deviation introduced by inaccurate coordinate system calibration, thereby effectively improving the positioning accuracy. Based on the more accurate positioning results, alarm decisions are made, thereby effectively improving the alarm accuracy of industrial machinery operation safety monitoring.

[0110] In addition, in the above Figure 1Based on the given basic method, to further avoid false detections and improve alarm accuracy, when obtaining the three-dimensional position information of the human target involved in step 102, the bounding box corresponding to each detected human target can be further determined. Here, the detected bounding box needs to be associated with the segmentation mask, that is, associated with a certain human target, i.e., the bounding box belongs to a certain human target. At the same time, after determining that a detected human target (hereinafter referred to as the first human target) is located inside or near the fence (for example, the human target is located within a set distance from the fence), after determining the positioning result of the human target inside or outside the fence and before determining whether to trigger an alarm, a process for identifying false detection targets is added, and the decision on whether to trigger an alarm is further based on the result of the false detection target identification process. The specific processing can be as follows:

[0111] Step a: Input the human target bounding box corresponding to the first human target and the image in which it is located into the pre-trained false detection removal model;

[0112] Step b: In the false detection removal model, feature extraction is performed based on the input image and human target bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection removal base library to determine whether the current frame of the first human target is a false detection target and its confidence level.

[0113] The process involves storing information on registered false positive targets generated through a registration process in a false positive detection database. Specifically, the user-inputted false positive target bounding box and its corresponding image are input into the false positive detection model. Feature extraction is performed on the bounding box and image to generate an embedding vector, which, along with the bounding box information, is stored in the database as registered false positive target information. During the inference process of the false positive detection model, feature extraction is performed on the bounding box and image of the first detected human target (using the same processing method as for registered targets) to generate an embedding vector. This vector is then compared with the registered false positive target information in the database to determine whether the first human target is a false positive target and to obtain the corresponding confidence level. This allows for the identification of false positive targets. Compared to existing technologies that use classification models to identify false positive targets, this method, when introducing new false positive targets in a new scenario, only requires registration of the false positive targets and does not require model training to achieve good false positive identification results. It allows for rapid response and identification of false positive targets in new work scenarios.

[0114] Meanwhile, to further improve the false detection accuracy of the false detection removal model, when obtaining the 3D position information of the human target involved in step 102, at least one specified part target box (e.g., face box and / or head and shoulder box) corresponding to each detected human target can be further determined. Here, the detected target box also needs to be associated with the segmentation mask, that is, associated with a certain human target, and in fact, associated with other target boxes of the same human target, that is, these target boxes belong to a certain human target. At the same time, in step a, at least one specified part target box of the first human target is further input into the false detection removal model. In step b, feature extraction is further performed based on at least one specified part target box, and binary classification is performed based on the feature extraction results to predict whether the first human target in the current frame is indeed a human target. Among them, the binary classification can be based on the relative position constraints between multiple target boxes belonging to the same human target to further improve the accuracy of binary classification, thereby effectively improving the accuracy of non-human targets (i.e., false detection targets).

[0115] In addition, a human body tracking model can be used to perform target detection processing on 3D point cloud images. For each human target detected in the 3D point cloud image, a corresponding segmentation mask can be output. The REID features of the human target can be further determined, and the trajectory information of the human target can be determined through trajectory tracking to further improve the accuracy of human target detection and optimize the problem of missed human target detection, especially in cases of target occlusion, multiple postures such as squatting and bending over, and fewer human features such as wearing a one-piece solid color electrostatic clothing.

[0116] Specifically, the human body tracking model may include a multi-task detection sub-model and a tracking sub-model;

[0117] In the multi-task detection sub-model, target detection processing can be performed on the acquired 3D point cloud image, a segmentation mask can be determined for each detected human target, and the corresponding human target box and at least one target box of a specified part can be predicted; REID features for personnel tracking can be determined based on the human target box and at least one target box of a specified part.

[0118] In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to a target (hereinafter referred to as the first target) output by the multi-task detection sub-model are received. The features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

[0119] Because the above human tracking model incorporates trajectory tracking of detected human targets, it can still predict target information in the current frame based on trajectory tracking results, combined with the target segmentation mask, bounding box information, and feature information from previous frames, even if no human target is detected in the current frame due to situations such as target occlusion, multiple postures such as squatting and bending over, or few human features due to wearing a solid-color electrostatic suit. This improves the problem of missed detection caused by the above situations.

[0120] The method described in this application can be implemented via a display and, in conjunction with an image data acquisition device and a processor, constitutes a monitoring system for industrial machinery operation safety. The image data acquisition device is used to collect data at the work site, and the processor is used to determine the three-dimensional alarm area, detect human targets, determine their location information, and generate alarm signals.

[0121] The following specific embodiments illustrate the detailed implementation of the monitoring method for industrial machinery operation safety in this application.

[0122] Figure 2 This is a schematic diagram illustrating the specific process of the monitoring method for industrial machinery operation safety in a particular embodiment of this application. Figure 3 This is a schematic diagram of a security monitoring system in a specific embodiment. The methods in the following embodiments are described from the processing perspective of each entity in the system, wherein a stereo vision camera is used as an example to illustrate the image data acquisition device. Figure 2 and Figure 3 As shown, the method specifically includes:

[0123] Step 201: The display receives the configuration of the ground alarm zone of the industrial machinery and the configuration of the fence posts of the industrial machinery and sends them to the processor.

[0124] As mentioned earlier, the configuration of fence posts for industrial machinery can be automatically obtained through data acquisition and analysis by a stereo vision camera, or it can be input by the user. In this embodiment, the fence post configuration can be determined by the display receiving user input. Simultaneously, the display can also receive user input of ground alarm zone configuration and advanced safety monitoring parameters, such as alarm sensitivity and target height, for adjusting the alarm decision conditions in subsequent safety prevention alarm modules. The display can receive user input during system initialization and can send relevant user input to the processor to complete the configuration of relevant conditions. The ground alarm zone configuration can be as follows: Figure 4 As shown, this is a polygonal regular line drawn on the solid ground area. Figure 4 The red line in the diagram represents the ground area within the fence; the fence post configuration refers to the arrangement of the fence posts and crossbars, which can be marked with straight lines. Figure 4Lines ① and ② in the diagram represent fence posts. In practice, the fence posts configured in the system should preferably be vertical posts, followed by horizontal bars parallel to the ground. If neither of these is available, diagonal bars or similar structures should be selected, but it must be ensured that they are within the plane corresponding to the fence.

[0125] Step 202: The processor acquires a 3D point cloud image of the work site in the current frame.

[0126] In this embodiment, a stereo vision camera collects data at the work site. Based on this collected data, a two-dimensional image can be generated, and the depth information of each coordinate point in the two-dimensional image can be calculated. Combining the two-dimensional image and the depth information together yields the position and image information (e.g., RGB values) of each point in the three-dimensional point cloud. This combination of information is called a three-dimensional point cloud image. The process of obtaining a three-dimensional point cloud image based on the collected data is called the process of determining the three-dimensional point cloud image. The process of determining the three-dimensional point cloud image can be completed in the stereo vision camera or in the processor. This step can be implemented using existing methods, which will not be elaborated here.

[0127] In addition to the implementation method of this embodiment, if the image data acquisition device is a radar camera, two-dimensional image data can be acquired through the camera, and depth information can be acquired through radar. The two-dimensional image and depth information are then fused to obtain a three-dimensional point cloud image. The process of obtaining a three-dimensional point cloud image based on the data acquired by the radar camera is called the process of determining the three-dimensional point cloud image. This process of determining the three-dimensional point cloud image can be completed either within the radar camera or in the processor.

[0128] Step 203: The processor uses the multi-task detection model in the human body tracking model to process the 3D point cloud image, and outputs the corresponding segmentation mask and three target boxes for each human target detected in the 3D point cloud image.

[0129] In this embodiment, the human tracking model includes a cascaded multi-task detection sub-model and a human tracking sub-model. This step utilizes the multi-task detection sub-model to detect human targets, calculating the REID features, segmentation mask, and bounding boxes of the human target. In this embodiment, three bounding boxes are output and associated with the human target: the human bounding box, the head and shoulder bounding box, and the face bounding box. Each output bounding box is associated with the detected human target and its segmentation mask.

[0130] The specific multi-task detection sub-model can be implemented using various model structures; this embodiment uses the transformer structure. For example... Figure 5The diagram shows the structure of the multi-task detection sub-model in this embodiment. This sub-model includes a backbone network, an encoder, and a decoder. The backbone network can use basic backbone networks such as ResNet or Swin Transformer for feature extraction, and can be adapted to different hardware platforms and requirements. The encoder fuses the multi-scale features output by the backbone network to unify the feature scale. The decoder implements model computation functions, including calculating segmentation masks, bounding boxes, and REID features. In the decoder, as shown... Figure 5 As shown, the rectangle represents convolution calculation, the ellipse represents convolution kernel update calculation, which is the attention mechanism, and the triangle represents normalization operation.

[0131] In existing object detection methods, the output bounding boxes are usually independent of each other. However, in this embodiment, the output bounding boxes need to be associated with the segmentation mask, that is, associated with a specific target. To achieve this association, the decoder in this embodiment uses an attention mechanism to predict the target category and generates a corresponding convolutional kernel for each target. Based on the convolutional kernel corresponding to the first target (classified as human), the first target is segmented to obtain a segmentation mask. Based on the convolutional kernel corresponding to the first target, the human bounding box and at least one specified body part bounding box are predicted by predicting the corner heatmap of the bounding box. The segmentation mask corresponding to the first target is multiplied with the image features of the current frame to obtain the REID feature of the first target. As can be seen from the above decoding process, the acquisition of the target segmentation mask, target bounding box prediction, and REID feature are all based on the convolutional kernel corresponding to the target. Therefore, the predicted output bounding boxes and segmentation masks are naturally correlated and both correspond to the targets corresponding to the respective convolutional kernels.

[0132] More in detail, such as Figure 5 As shown in the decoder, the specific computational processing includes three branches. The first branch (also known as the target instance branch) predicts the target using an attention mechanism network. For each target (specifically, a Proposal or a Query), the network dynamically generates a corresponding convolutional kernel. Additionally, this branch predicts the target instance's category and confidence score; in other words, it determines which targets belong to the human body.

[0133] The second branch (also known as the segmentation branch) performs convolution / matrix multiplication operations between the convolutional kernel corresponding to each target instance and the high-resolution global feature map. This directly generates a binary mask image (Mask Logits), which, after sigmoid activation, becomes the predicted mask for that target instance (i.e., the aforementioned segmentation mask). The loss function for both the first and second branches during training can consist of two parts: classification loss (such as Focal Loss) and mask loss (such as Dice Loss and Binary Cross-Entropy).

[0134] The third branch (also known as the detection branch) extracts regression features, resulting in three bounding boxes for each human target. The human body bounding box, head and shoulder bounding box, and face bounding box are calculated using the same kernel and are inherently related. Since this embodiment outputs three bounding boxes, their calculation is achieved by predicting a heatmap of the target box corners. In this embodiment, the heatmap is treated as a probability distribution or weight distribution, and continuous, accurate coordinates are obtained by calculating the expected value (mean) of this distribution. Specifically, for a target (hereinafter referred to as the first target), based on the convolution kernel corresponding to the first target, the probability of each target point being a corner point of any bounding box is predicted. This probability is used as the weight of the corresponding target point, and the weighted average of the coordinates of each target point is calculated. The result is used as the corner coordinates of any bounding box; where any bounding box is one of the human body bounding box, head and shoulder bounding box, or face bounding box.

[0135] For example, taking the calculation of a corner point 'a' in the bounding box as an example, the value of f(x,y) represents the probability that the coordinate point (x,y) is corner point 'a'. The coordinate position corresponding to the peak region of f(x,y) should be the corner point position. In this embodiment, the probability value is used as the weight to calculate the corner point position using a weighted average, which can locate the corner point more precisely. The calculation formula is as follows: average horizontal coordinate (i.e., the horizontal coordinate of corner point 'a'):

[0136]

[0137] Average ordinate (i.e., the ordinate of corner point a):

[0138]

[0139] Here, it is assumed that the size of the Heatmap is N. (B 2) (H / 8) (W / 8) represents the probability distribution corresponding to the first target for the feature map after 8x downsampling, where the original image size is H. W and N are the batch size, and B is the number of bounding boxes (3 bounding boxes in this example, i.e., B=3). Each bounding box corresponds to 2 corner points. The coordinates of B bounding boxes are calculated and output, with a size of N. B 4. Each target box is represented by 4 values: x1, y1, x2, and y2, which represent the coordinates of the top left and bottom right corners of the target box, respectively.

[0140] During model training, the heatmap of the third branch uses the Focal Loss loss function, where the ground truth is a map generated by simulating the Gaussian distribution of the target corner points. Furthermore, the above method can be easily extended to human keypoint detection, such as adding keypoints for human hands, feet, and heads, to achieve precise localization of target parts.

[0141] like Figure 5 As shown, after the second branch is processed, a REID branch is connected, and the feature vector is calculated using matrix multiplication. Then, after post-processing (i.e., other processing by the model), the refined extraction of the target REID features is achieved. The main processing steps for REID feature extraction can be as follows: Figure 6 As shown, the segmentation mask of the target is multiplied by the image features.

[0142] To enhance the discriminative power of REID features among different targets, the REID branch employs contrastive learning to train the model for REID feature extraction. Specifically, during training, the target embedding features of the input reference frame and the current frame are compared. Based on similarity, the m1 closest and m2 furthest targets are selected to calculate the triplet loss and cosine embedding loss. For a set of query and reference features, the similarity between all possible sample pairs is calculated, encouraging positive sample pairs to have significantly higher similarity than negative sample pairs. Therefore, during model training, each input requires an image pair, specifically two images of the same tracking ID target X in different frames. For target X, the portion of target X in both images constitutes a positive sample pair, while the portion of target X in one image and the portion of other targets (not X) in the other image constitute a negative sample pair.

[0143] The above describes the processing of the multi-task detection sub-model.

[0144] Step 204: The processor uses the tracking sub-model in the human body tracking model to determine the trajectory information of the human target and the updated target bounding boxes, and sends them to the display.

[0145] In the tracking sub-model of this embodiment, the human body bounding box, head and shoulder bounding box, face bounding box and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human body bounding box and at least one specified part bounding box are determined.

[0146] The tracking sub-model can adopt existing tracking model architectures, including steps such as feature updating, similarity calculation, trajectory association, and confusion analysis. Feature updating can include IOU features, shape features, Euclidean distance features, area features, velocity features, and REID features. Trajectory association can employ the commonly used Hungarian association method, and feature fusion can utilize various combinations such as weighted feature addition and multiplication. Confusion analysis can target two highly similar, easily confused targets, combining multiple features to associate or disconnect them.

[0147] Similarity calculation can be performed using existing methods, such as calculating feature similarity using cosine distance. In this embodiment, to further improve the accuracy of similarity calculation, a bidirectional softmax distance calculation method is adopted. Bidirectional softmax is a bidirectional matching method in the embedding space. Taking the similarity calculation between the first target and the first historical trajectory as an example, the bidirectional softmax distance calculation method is as follows:

[0148] Calculate the distance between the first target and the first historical trajectory, and refer to it simply as the first distance;

[0149] Using the first target as the center, calculate the first matching probability between the first target and all historical trajectories;

[0150] Using the first trajectory as the center, calculate the second matching probability between the first trajectory and each human target in the current frame;

[0151] The average of the first matching probability and the second matching probability is used as the similarity between the first target and the first historical trajectory.

[0152] For example, suppose there are N detected objects in frame t, with corresponding embedding features n, and M matching candidate objects (i.e., M historical trajectories) in the historical trajectory, with corresponding embedding features m. The similarity f(i,j) between the current detected object i and the matching candidate j is obtained through bi-softmax:

[0153]

[0154] The first part involves matching the detected object to the matching candidate object, using the feature vector of the current detected object i. Centered on the M historical trajectories, calculate the probability of matching it with all M historical trajectories (i.e., the first matching probability), and add two parameters. and (For example, detection confidence) to adjust the original similarity between the target and historical trajectories; the second part is matching candidate objects to detected objects to match the feature vectors of candidate object j. Centered on the first target, calculate the matching probability (i.e., the second matching probability) between the first target and the N detected objects in the current frame. Finally, average the first and second matching probabilities.

[0155] The bidirectional softmax method described above calculates the bidirectional similarity between the target and the trajectory. The first part measures the distance from the current target to each historical trajectory, while the second part measures the distance from the current historical trajectory to all targets. Newly appearing targets, disappearing trajectories, or falsely reported targets are difficult to match bidirectionally, resulting in low similarity scores and matching scores, making it impossible to find a match. This effectively identifies false alarms and newly appearing targets. For targets without a matching historical trajectory, if the detection confidence is high, they can be identified as newly appearing targets, and new tracking can begin; if the detection confidence is low, they are considered false alarms.

[0156] Step 205: The display acquires the trajectory information of the human target and the updated target bounding boxes and displays them.

[0157] The display receives the results of the processor's target detection and tracking, namely the trajectory information of the human target and the updated bounding boxes, and displays the corresponding position information of the human target on the interface. Specifically, both the trajectory information and the bounding boxes can be displayed, or only the bounding boxes can be shown.

[0158] Step 206: The processor determines the three-dimensional alarm zone of the industrial machinery based on the received ground alarm zone configuration and fence post configuration, and sends it to the display.

[0159] The specific processing for determining the three-dimensional alarm zone in this embodiment may include:

[0160] Determine regular polygonal regions on the ground in the world coordinate system based on the configuration of ground alarm zones;

[0161] The projection points from the straight line containing the fence posts to the ground are determined based on the fence post configuration, and the vertices of the polygonal regular area on the ground are updated based on the projection points.

[0162] The ground plane X is obtained by fitting the vertices of the regular polygonal region on the ground. All vertices are then projected onto the ground plane X to generate the edges of the ground plane.

[0163] The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

[0164] The more detailed implementation steps are as follows:

[0165] Step 1: Obtain the regular polygonal region on the ground.

[0166] 1) Convert the configured ground alarm area into a polygon, and convert the polygon vertices from image pixel coordinates to 3D coordinates in the world coordinate system. Specifically, for each polygon vertex pixel coordinate (u_i, v_i), calculate the 3D point through binocular matching, obtain its world coordinates, calculate the disparity based on the left and right eyes, and then convert the disparity into depth and point cloud.

[0167] 2) Obtain the initial ground polygon P_base = {(x_i, y_i, z_i) | i=1,2,...,n}, where i is the vertex index of the polygon, and z_i should be 0. Check the Z coordinates of all vertices to confirm that they are close to 0 (ground points), and ensure that the vertices are arranged in clockwise or counterclockwise order.

[0168] Step 2: Straight line processing and 3D fitting of fence posts.

[0169] 1) Obtain the pixel coordinates of the fence posts: Extract the pixel endpoint coordinates of each post line segment labeled by the user. For each line segment, sample multiple intermediate points to improve accuracy. For each pixel (u, v), obtain the corresponding 3D point coordinates to obtain the 3D point set Q_j corresponding to each post.

[0170] 2) Multi-directional line fitting: For each set of pillar points Q_j, fit a 3D line using RANSAC or least squares methods. The line model can be represented as... ,in Let be the base point, and d be the direction vector; for vertical pillars, d should be close to (0,0,1); for horizontal / diagonal pillars, d can be in any direction. Calculate the line parameters and fitting error, and optimize the line position.

[0171] 3) Calculation of Ground Projection Points: Find the intersection point of the fitted line of the fence post and the ground plane (Z=0), and use this intersection point as the ground projection point. The equation of the fitted line of the fence post is: Let z = 0, then we get (When d_z ≠ 0). Projection point: If d_z ≈ 0 (i.e. the line is parallel to the ground), then the line is projected perpendicularly onto the ground.

[0172] In addition, to more accurately determine the 3D alarm zone, the handling of abnormal columns can be further enhanced. Theoretically, columns projected onto the ground from the same fence surface should lie on the same straight line. If the distance between the straight lines projected onto the ground from the same fence surface is found to be large, then an abnormal column may exist. Based on the relationship between multiple columns on that fence surface and the relationship between the projection and the ground polygon, if the projected line of a certain column is far from the projected lines of other columns or far from the edge of the ground polygon, then the column is considered to have a potential drawing problem and is treated as an abnormal column, not used in subsequent processing, i.e., not used to generate the 3D alarm zone, and the user is prompted to confirm and modify it.

[0173] Step 3: Ground polygon correction and ground equation fitting.

[0174] 1) Fence post projection point integration: For each post's ground projection point p_proj, find the nearest edge E_k in the polygon (composed of vertices p_k and p_{k+1}), and calculate the perpendicular projection point from p_proj to edge E_k. If the projection point is located on a line segment (not near an endpoint), insert the projection point as a new vertex; if the projection point is close to an existing vertex, replace the existing vertex. In this way, the boundary of the ground alarm area can be effectively completed when it is obscured.

[0175] 2) Polygon optimization: Remove duplicate vertices and non-endpoint vertices among collinear vertices, ensure that the vertex order remains consistent (clockwise / counterclockwise), and check and fix self-intersections.

[0176] 3) Ground Equation Fitting: Fit the ground plane equation using all the corrected polygon vertices, specifically using the least squares method, represented as: z = f(x,y) = a'x + b'y + c'; calculate the fitting error and evaluate the ground flatness. Then, project all polygon vertices onto the fitted ground plane to ensure that the polygons are accurately located on the ground.

[0177] Step 4: Generate a three-dimensional alarm zone and improve the accuracy of the fence boundary by using the fence post information. At the same time, process the situation of posts in various directions to generate a more accurate three-dimensional alarm zone.

[0178] 1) Top surface generation: Copy the vertices of the bottom polygon, set the Z coordinate to the camera height H_cam, P_top = {(x_i,y_i, H_cam) | (x_i, y_i, z_i) ∈ P_base}, thus obtaining a top surface that is parallel to the ground and at the camera height.

[0179] 2) Side Generation: For each edge E_i of the base (the edge connecting p_i and p_{i+1}), create a side quadrilateral. The four vertices of the quadrilateral are: p_i → p{i+1} → (p{i+1} + (0,0,H_cam)) → (p_i + (0,0,H_cam)). All sides form the perimeter wall of the 3D alarm area. Based on this, retain the 3D points obtained when fitting the original column lines; that is, the actual fence area is a set of 3D points containing noise.

[0180] Through the above processing, an accurate three-dimensional alarm zone can be obtained.

[0181] Step 207: The display shows the 3D alarm area on the interface.

[0182] Step 208: The processor determines the effective three-dimensional points of the first target in the current frame based on the trajectory information and segmentation mask of the first target, and determines the positioning result of the corresponding human target inside and outside the fence in the current frame based on the relative positional relationship between the effective three-dimensional points and the three-dimensional alarm area.

[0183] Based on the trajectory information and segmentation mask of the first human target, the effective 3D points of the first target in the current frame are determined. Specifically, based on the trajectory information, occluded human target points in the segmentation mask can be added or redundant human target points can be deleted. Furthermore, in this embodiment, the detection and tracking processing of the human tracking model can be performed first based on a 2D image. After obtaining the results corresponding to the 2D pixels, the depth information corresponding to those results is then added to obtain the 3D points.

[0184] After determining the valid three-dimensional points of the first target, the relative positional relationship between each three-dimensional point of the target and the three-dimensional alarm area is determined point by point, and the positioning result of the first target in the current frame inside or outside the fence is calculated. The positioning result may be inside the fence, outside the fence, or near the fence (i.e., within a set distance range from the fence). Typically, being inside the fence means being inside the fence and having a distance greater than a set first distance threshold from the fence boundary; being outside the fence means being outside the fence and having a distance greater than a set second distance threshold from the fence boundary. The first and second distance thresholds can be set according to needs or alarm sensitivity, and they can be equal or unequal. Along with the positioning result, the confidence level of the corresponding positioning result, i.e., the positioning confidence level, can also be calculated.

[0185] As shown above, in this embodiment, the relative positional relationship between the effective three-dimensional points of the human target and the three-dimensional alarm area is compared to determine the positioning result of the human target being inside or outside the fence, thereby effectively reducing the positioning error.

[0186] In industrial production line environments, false alarms reduce production efficiency, thus requiring high alarm accuracy. Therefore, as mentioned above, this application performs false target identification processing on detected human targets located inside or near fences to further improve alarm accuracy. In this embodiment, this is performed through the following steps 209-210.

[0187] Step 209: The processor uses a false detection removal model to perform binary classification on the first human target detected in step 208 that is located inside or near the fence, to determine whether it is a human target.

[0188] The false detection model in this embodiment performs two methods of false detection identification. This step performs the first method, which determines whether a detected target that may trigger an alarm is indeed a human target. This involves binary classification of the target as human or non-human. If a target is determined to be a non-human target and the classification confidence level is higher than a set classification confidence threshold (meaning the target's classification as non-human is highly credible), then it can be determined that no alarm signal will be output for that target. The specific binary classification process can be implemented using various existing methods. Specifically, during binary classification, constraints can be set based on three bounding boxes associated with the same target to distinguish between human and non-human targets. For example, the constraints could be: for a human target, the head and shoulder bounding box should include the face bounding box; and the human target bounding box should include both the head and shoulder bounding box and the face bounding box.

[0189] Step 210: The processor uses the false detection removal model to compare the embedding vector of the human target bounding box of the first human target with the registered false detection target information stored in the false detection base library to determine whether the first human target is a false detection target in the current frame.

[0190] This step performs the second method of false detection identification, which is achieved through retrieval.

[0191] Specifically, firstly, a false detection base library and its registered false detection target information are generated based on the false detection target information input by the user. More specifically, the system can receive the false detection target bounding boxes and their corresponding images input by the user, extract the false detection target images from the images, perform feature extraction and post-processing to generate embedding vectors for the corresponding false detection targets, and register them in the false detection base library. The false detection base library stores the registered false detection target information, which specifically includes: the target bounding box information of the false detection targets and the corresponding embedding vectors.

[0192] When identifying a false positive for a first human target, the bounding box of the first human target and its surrounding image are input into the false positive model for inference. An image of the first human target is extracted from the image, and after feature extraction and post-processing, an embedding vector for the first human target is generated. This embedding vector is compared with the registered false positive target information in the false positive database. If the similarity between the embedding vector and any registered false positive target's embedding vector exceeds a set threshold, the first human target in the current frame is determined to be a false positive, and the corresponding confidence level is determined. If a target is identified as a false positive and the retrieval confidence level is higher than the set retrieval confidence level threshold (meaning the target is highly likely to be a false positive), then it can be determined that no alarm signal will be output for that target.

[0193] In steps 209 and 210 above, a false detection removal model is used to achieve binary classification of human targets and retrieval of falsely detected targets for comparison, thereby realizing the identification of falsely detected targets. In actual processing, binary classification can be implemented using a classification model, and retrieval and comparison can be implemented using a retrieval model. In this embodiment, to improve computational efficiency, the classification and retrieval models are merged into a single classification model with retrieval functionality, i.e., a false detection removal model.

[0194] The false positive removal model consists of a backbone network and two branch networks. The backbone network extracts image features. The first branch network performs binary classification based on the image features extracted by the backbone network. The second branch network retrieves and compares false positive targets based on the image features extracted by the backbone network and the registered false positive target information in the false positive removal database. During training, the first branch network performs binary classification on the extracted image features, using a first loss function to minimize the difference between the predicted probability distribution and the true distribution. The second branch network maps the extracted image features to an embedding space, using a second loss function to optimize the embedding space, causing similar targets to cluster and dissimilar targets to separate. The first loss function can be cross-entropy loss, and the second loss function can be contrastive loss or triplet loss. To train both tasks simultaneously, a joint loss function is defined to optimize the parameters of the false positive removal model. The joint loss function could be, for example: .

[0195] Here, λ is a hyperparameter used to balance the losses of the two tasks. The cross-entropy loss of the classification task could be, for example,... Where yi is the real label. It represents the probability corresponding to the binary classification output;

[0196] The triplet loss for a retrieval task could be, for example: Where a is the anchor sample, p is the positive sample, n is the negative sample, d is the Euclidean distance, α is the margin, representing a bias parameter, and N is the batch size.

[0197] To better train the aforementioned model, samples need to be selected appropriately within each batch. In this embodiment, a hybrid sampling strategy is used to select training samples, preparing two sub-batches according to the batch construction methods for the classification and retrieval tasks respectively:

[0198] • Sub-batch A: Selects typical training image samples for classification. open;

[0199] • Sub-batch B: Selects typical training image samples for retrieval. Zhang, these samples need to be from different categories.

[0200] Two sub-batches are concatenated into a larger batch and fed into the model. For sub-batch A, the backbone network and the first branch network are used to process sub-batch A and calculate the value of the first loss function. For sub-batch B, the backbone network and the second branch network are used to process sub-batch B and calculate the value of the second loss function. The joint loss function is calculated based on the values ​​of the first and second loss functions, and the model parameters are updated based on this value.

[0201] As described above, false positive targets are identified through steps 209 and 210, and the confidence level of the identification result is calculated.

[0202] Step 211: The processor determines whether to trigger an alarm and sends an alarm signal to the display when the alarm needs to be triggered.

[0203] In this embodiment, each detected human target can be individually assessed to determine whether it meets the alarm conditions. Once a human target that meets the alarm conditions is detected, an alarm is triggered. For any detected human target, the alarm can be determined based on the localization results of multiple frames, the corresponding confidence scores, the confidence scores of the three target boxes, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information. The aforementioned steps primarily obtain the model output results and corresponding confidence scores of the human target for the current frame. In this step, the alarm decision can be made by combining the model output and confidence scores of the same human target across multiple frames.

[0204] Optionally, different areas can be classified for detection. Specifically, the camera's field of view can be divided into multiple different areas based on the three-dimensional alarm zone, and different alarm strategies can be set for each area. For example, the first area can be the area near a fence, specifically, the area within a set range (e.g., the distance from the fence inside the fence is less than or equal to a first distance threshold, and the distance from the fence outside the fence is less than or equal to a second distance threshold); the second area can be the area inside the fence and the distance from the fence is greater than the first distance threshold; the third area can be the area outside the fence and the distance from the fence is greater than the second distance threshold. Different alarm strategies can be used in these three different areas to simplify system processing as much as possible while ensuring alarm accuracy.

[0205] For example, if the detected human target is located in the first region, the system configuration and the output and confidence level of the human target in the aforementioned models can be combined to determine whether the alarm conditions are met. If the detected human target is located in the second region and the location confidence level is greater than the set location confidence level threshold, and it is not determined to be a false detection target after the processing in steps 207 to 208, the alarm conditions can be directly determined to be met and the alarm can be triggered. If the detected human target is located in the third region, the human target is directly determined not to meet the alarm conditions.

[0206] For detected human targets located in the first region and human targets located in the second region but with a location confidence level not exceeding a set location confidence threshold, the system configuration, along with the target's output and confidence levels across multiple models, are used to jointly determine whether an alarm condition is met. Specifically, in determining whether an alarm condition is met, this embodiment can be based on information from the following two dimensions:

[0207] • External parameter input: During the system initialization configuration in step 101 above, configuration parameters input by the user can be received, thereby providing user-customizable dynamic threshold and percentage condition settings. It supports flexible expansion of judgment logic according to actual application scenarios, including alarm sensitivity, the confidence height area of ​​human targets, etc., and affects the value of confidence level, etc. through corresponding parameters.

[0208] • Spatial location determination: The positioning results and their confidence levels output by the human body tracking model are used as the core judgment criteria, and combined with the output results of the false detection elimination model, the accuracy of positioning and intrusion determination is improved.

[0209] More specifically, when determining whether alarm conditions are met, a hierarchical filtering mechanism can be employed. This involves multi-frame smoothing of certain status indicators to enhance the system's anti-interference capability and robustness. The feature indicators involved in the final decision on whether alarm conditions are met may include:

[0210] • Confidence levels of the three bounding boxes;

[0211] • Segmentation confidence, mask area, etc. of the segmentation mask;

[0212] • Target tracking trajectory information status, tracking confidence level, and continuous tracking duration, etc.;

[0213] • The target classification confidence score and the number of classification categories in multiple consecutive frames output by the human tracking model;

[0214] • De-false detection model to retrieve target confidence assessment to determine whether it is a false detection target;

[0215] • The positioning reliability of the target in the world coordinate system output by the human body tracking model;

[0216] • The number of effective 3D point clouds corresponding to the target and the proportion of point clouds located within the fence in the human body tracking model;

[0217] • Target estimation altitude information, etc.

[0218] The above process is used to determine whether an alarm should be triggered, and when it is determined that an alarm needs to be triggered, an alarm signal is sent to the display to notify the display to handle the alarm.

[0219] Step 212: After receiving the alarm signal, the display performs alarm processing.

[0220] Specific alarm handling could involve displaying an alarm icon on the interface and playing an alarm prompt sound.

[0221] Step 213: The processor counts the human targets in the closed blind zone and sends the counting results to the display.

[0222] In addition to the alarm decision method given in step 211 above, this embodiment also provides a closed blind spot alarm function, which can provide special handling for situations where a camera cannot cover a complete area and a visual blind spot occurs.

[0223] First, define the meaning of a closed blind zone.

[0224] Within an industrial machinery enclosure, an area that meets the following conditions is considered a closed blind spot:

[0225] • The target must be able to enter or leave the area only through the camera's field of view (the area covered by the camera and unobstructed); for example, there is a fence around the robotic arm, and personnel cannot enter or leave the fence through areas not visible to the camera, but can only enter or leave the fence through areas visible to the camera.

[0226] • The ground in this area is not visible in the image. There is a portion of the ground that should belong to the alarm area, but it is inconvenient to configure it into the ground alarm area.

[0227] Since cameras can generally only be mounted on brackets above the fence, they may not completely cover the entire ground area within the fence, resulting in the aforementioned blind spots. Targets within these blind spots will miss alarms. For example, the camera may only capture part of the boundary line of the ground alarm area; the area near the missing boundary line is a blind spot. When personnel crouch down or get close to a corner of the fence that is not visible to the camera, a target may miss the alarm.

[0228] For situations involving closed blind zones, the blind zone alarm function described in this step can be used to count human targets located within the closed blind zone in a three-dimensional coordinate system. Specifically, the count can be incremented by 1 when a human target enters the closed blind zone and decremented by 1 when a human target leaves the closed blind zone. This count result represents the number of people currently in the closed blind zone. Alternatively, the number of people entering the closed blind zone can be incremented by 1 when a human target is detected entering the closed blind zone and the number leaving the closed blind zone can be incremented by 1. The difference between the number of people entering and leaving the closed blind zone can be used to obtain the number of human targets within the closed blind zone. When the number of human targets within the closed blind zone is not zero, an alarm can be triggered, generating an alarm signal and sending it to the display. The detection of entering and leaving the closed blind zone can be achieved through line-crossing detection, and the counting of human targets within the closed blind zone can be achieved through line-crossing counting. The line-crossing rule line can be the boundary line of a reused alarm rule area, for example... Figure 7 This is a schematic diagram of the cross-line rule line. The red line is the boundary line of the alarm rule area (i.e., the edge of the aforementioned ground polygon). The yellow solid line close to the bottom edge of the image is the cross-line rule line. Considering that the closed blind zone usually appears in the area below the camera, that is, the off-screen area at the bottom of the screen, in this embodiment, it is assumed that the upper left corner of the screen is the origin of the coordinate system, the right is the positive X-axis, and the down is the positive Y-axis. Then, two points with the maximum and second maximum vertical coordinate Y are selected from the points of the ground polygon, and the line connecting these two points is taken as the cross-line rule line.

[0229] Step 214: The display receives and displays the human target count results of the closed blind zone sent by the processor.

[0230] After processing in step 203, the processor determines the human target count within the closed blind zone and sends it to the display. In this step, the display receives the count result from the processor and displays it on the interface in real time. When the count result is 0, if the processor sends an alarm signal, the display can perform an alarm operation; alternatively, the display can also perform an alarm operation automatically when it determines that the received count result is 0.

[0231] The closed blind zone alarm function in steps 213 and 214 is an extended function. For shooting points without closed blind zones or where closed blind zones do not need to be considered, users can choose to disable the closed blind zone alarm function through the display interface. After receiving the user's selection, the display sends the closed blind zone disabling information to the processor, notifying the processor to stop the human target counting processing in the closed blind zone. Simultaneously, for other obstructed closed areas, this function can also be enabled when the entrance / exit is completely unobstructed to prevent long-term obstruction by staff from causing missed alarms and potential safety accidents. In the event of a false closed blind zone alarm, the system can be reset to zero through the display interface after manually confirming the safety of personnel in the area.

[0232] This concludes the method flow in this embodiment.

[0233] As can be seen from the above processing described in this application and specific embodiments, in this application, a three-dimensional alarm area can be obtained by fitting the three-dimensional region of the alarm area boundary. Combined with the effective three-dimensional points of the human target generated by human segmentation calculation, the judgment of whether the target is inside or outside the fence can be realized in three-dimensional space through relative positional relationships, which can effectively improve the positioning accuracy of the human target and thus improve the alarm accuracy. At the same time, in view of the problem of poor detection of human targets in this case, this application adopts a human tracking model of target detection and tracking. Target detection achieves more accurate human pixel positioning, and trajectory tracking effectively improves the target detection rate in cases of occlusion and multiple poses. Furthermore, a false detection removal model can be used to realize the binary classification function of human body and non-human background based on classification task and registration retrieval task, effectively identify false detection targets, improve the false alarm removal effect, and reduce the amount of computation. In summary, through multiple processing methods, the alarm accuracy of security monitoring can be effectively improved, and the processing efficiency can be increased.

[0234] The above describes the specific implementation of the monitoring method for industrial machinery operation safety in this application. As mentioned earlier, this application also provides a monitoring system for industrial machinery operation safety, including: an image data acquisition device, a processor, and a display.

[0235] Among them, the image data acquisition equipment is used to collect data at the work site;

[0236] The processor is configured to determine the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device based on the configuration of the ground alarm area and the fence post configuration of the industrial machinery; it is also configured to acquire a three-dimensional point cloud image of the work site in the current frame; and acquire the three-dimensional position information of the human target detected in the three-dimensional point cloud image in the current frame; it is also configured to determine the positioning result of the corresponding human target inside and outside the fence in the current frame based on the three-dimensional position information of the human target and the three-dimensional alarm area; and determine whether to trigger an alarm based on the positioning result, and notify the display when it is determined that an alarm has been triggered; wherein, the three-dimensional point cloud image is a combination of two-dimensional image and depth information determined based on the acquired data;

[0237] The display is used to acquire and display the three-dimensional alarm area of ​​industrial machinery, as well as to acquire and display the position information of human targets appearing in the imaging field of view, and to perform alarm processing after receiving alarm signals.

[0238] Optionally, the processor may determine the location result of the corresponding human target inside or outside the fence, which may specifically include: determining the effective three-dimensional point of the human target based on the three-dimensional position information, and determining the location result of the corresponding human target inside or outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area.

[0239] Optionally, in the target detection unit, the method of acquiring three-dimensional position information may specifically include: using a pre-trained human tracking model to perform target detection and tracking on the three-dimensional point cloud image, and outputting the corresponding segmentation mask and trajectory information for each human target detected in the three-dimensional point cloud image;

[0240] In the processor, valid 3D points are determined based on trajectory information and segmentation masks.

[0241] Optionally, in the processor, the human tracking model may include a multi-task detection sub-model and a tracking sub-model;

[0242] In the multi-task detection sub-model, target detection processing is performed on the 3D point cloud image. For each detected human target, a segmentation mask is determined, and its corresponding human target bounding box and at least one specified part target bounding box are predicted. Based on the human target bounding box and at least one specified part target bounding box, REID features for personnel tracking are determined.

[0243] In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

[0244] Optionally, in the multi-task detection sub-model, the target category is predicted through an attention mechanism, and a corresponding convolutional kernel is generated for each target; the first target is segmented based on the convolutional kernel corresponding to the first target classified as human body, and the segmentation mask corresponding to the first target is obtained; based on the convolutional kernel corresponding to the first target, the human body target box and at least one specified part target box corresponding to the first target are predicted by predicting the target box corner heatmap; the segmentation mask corresponding to the first target is multiplied with the image features of the current frame image, and the REID feature of the first target is obtained after post-processing.

[0245] Optionally, in the processor, the similarity calculation method between the first target and the first historical trajectory may include:

[0246] Using the features of the first target as the center, calculate the first matching probability between the first target and all historical trajectories;

[0247] Using the features of the first trajectory as the center, calculate the second matching probability between the first trajectory and each human target in the current frame;

[0248] The average of the first matching probability and the second matching probability is used as the similarity between the first target and the first historical trajectory.

[0249] Optionally, the process of predicting the human body bounding box and at least one specified part bounding box corresponding to the first target in the target detection unit may specifically include:

[0250] Based on the convolution kernel corresponding to the first target, the probability of each target point being a corner point of any target box is predicted. The probability is used as the weight of the corresponding target point, and the weighted average of the coordinates of each target point is calculated. The calculation result is used as the corner point of any target box. Among them, any target box is one of the human body target box corresponding to the first target and at least one target box of a specified part.

[0251] Optionally, in the processor, after performing target detection on the 3D point cloud image, the processor further determines the human target bounding box corresponding to each human target; it is also used to input the human target bounding box corresponding to the first human target and the image in which it is located into a pre-trained false detection removal model after the localization unit determines that the current frame of the first human target is located within the fence or within a set range from the fence; in the false detection removal model, feature extraction is performed based on the input image and target bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection base library to determine whether the current frame of the first human target is a false detection target and its confidence level.

[0252] In the processor, it is further determined whether to trigger an alarm based on the first result and its confidence level;

[0253] The false detection base database stores the registered false detection target information, including: the target bounding box information of the registered false detection target input by the user, and the embedding vector of the registered false detection target obtained by processing the target bounding box of the registered false detection target and the image where the target bounding box is located using the false detection model.

[0254] Optionally, in the processor, after target detection is performed on the 3D point cloud image, it is further used to determine at least one designated part target box corresponding to each human target; it is also used in the false detection removal model to further perform feature extraction based on the at least one designated part target box, and to perform binary classification processing based on the feature extraction results to predict whether the first human target in the current frame is indeed a human target; and to further determine whether to trigger an alarm based on the prediction results and their confidence levels.

[0255] Optionally, the false detection removal model includes a backbone network for extracting image features, a first branch network for binary classification based on the extracted image features, and a second branch network for retrieval based on the extracted image features and the registered false detection target information;

[0256] When training the false detection removal model, the first branch network performs binary classification on the extracted image features and uses the first loss function to minimize the difference between the predicted probability distribution and the true distribution; the second branch network maps the extracted image features to an embedding space and uses the second loss function to optimize the embedding space, so that similar targets are clustered and dissimilar targets are separated.

[0257] The training data for the false positive removal model includes two groups of training data. The first group of training data is processed using the backbone network and the first branch network to calculate the value of the first loss function. The second group of training data is processed using the backbone network and the second branch network to calculate the value of the second loss function. The weighted sum of the values ​​of the first and second loss functions is then used as the value of the joint loss function to update the model parameters.

[0258] Optionally, the processing of determining the three-dimensional alarm zone of industrial machinery based on the ground alarm zone configuration and the fence post configuration in the processor may specifically include:

[0259] Determine regular polygonal regions on the ground in the world coordinate system based on the configuration of ground alarm zones;

[0260] The projection points from the straight line containing the fence posts to the ground are determined based on the fence post configuration, and the vertices of the polygonal regular area on the ground are updated based on the projection points.

[0261] The ground plane is obtained by fitting the vertices of the regular polygonal regions on the ground, and all vertices are projected onto the ground plane to generate the edges of the ground plane.

[0262] The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

[0263] The display is further configured to receive ground alarm zone configuration and / or fence post configuration input by the user through the configuration interface, and forward it to the processor;

[0264] or,

[0265] The processor is further configured to determine the ground alarm zone configuration and / or the fence post configuration based on the data acquired by the image data acquisition device.

[0266] Optionally, the processor is further configured to retain the three-dimensional points of the fitted posts when determining the straight line where the fence posts are located based on the fence post configuration when generating the fence side; if the fitted fence post is located outside the three-dimensional alarm area, then the fence post is ignored.

[0267] Optionally, in the processor, when generating the fence side, the three-dimensional points of the fitted posts are retained when determining the straight line where the fence posts are located based on the fence post configuration; if the fitted fence post is located outside the three-dimensional alarm area, the fence post is ignored.

[0268] Optionally, the process of determining the relative positional relationship between the corresponding human target and the three-dimensional alarm area based on the relative positional relationship between the effective three-dimensional points and the three-dimensional alarm area in the processor may specifically include:

[0269] The relative positional relationship between each valid 3D point and the side of the 3D alarm area is determined point by point, and the positioning result of the corresponding human target inside and outside the fence is determined based on the judgment results of all valid 3D points.

[0270] Optionally, the process in the processor for determining whether to trigger an alarm based on the location result may specifically include:

[0271] Based on the positioning results corresponding to multiple frames and the corresponding confidence scores, the confidence scores of the human target bounding box and at least one specified part target bounding box, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information, determine whether to trigger an alarm.

[0272] Optionally, when there is a closed blind zone at the work site, the processor is further used to count human targets located in the closed blind zone and send the counting result to the display; when the counting result is not 0, it determines to trigger an alarm and notifies the display.

[0273] The display is used to acquire and display the number of human targets located in the closed blind area when there is a closed blind area at the work site; when it receives a notification that the processor count result is not 0, it performs an alarm process.

[0274] Optionally, the image data acquisition device can be further used to determine a 3D point cloud image and send it to the processor;

[0275] or,

[0276] In the processor, the processing of acquiring a 3D point cloud image can specifically include: determining the 3D point cloud image based on the acquired data.

[0277] Alternatively, the image data acquisition device may be a stereo vision camera or a radar camera.

[0278] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the disaster recovery processing method for implementing a video surveillance system as described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium stores instructions, which, when executed by a processor, can perform the steps in the disaster recovery processing method as described above.

[0279] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0280] Figure 8 An electronic device is also provided for this application. For example... Figure 8 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0281] The electronic device may include a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 802 is executed, a method for monitoring the safety of industrial machinery operations can be implemented.

[0282] Specifically, in practical applications, this electronic device may also include components such as a power supply 803 and an input / output unit 804. Those skilled in the art will understand that... Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0283] The processor 801 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and calling data stored in the memory 802, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0284] Memory 802 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. Processor 801 executes various functional applications and data processing by running the software programs and modules stored in memory 802. Memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 802 may also include a memory controller to provide processor 801 with access to memory 802.

[0285] The electronic device also includes a power supply 803 that supplies power to the various components. This power supply can be logically connected to the processor 801 via a power management system, enabling functions such as charging, discharging, and power consumption management. The power supply 803 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0286] The electronic device may also include an input / output unit 804, which can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 804 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0287] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for monitoring the operational safety of industrial machinery, characterized in that, include: Acquire and display the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device; Acquire and display the location information of human targets appearing within the imaging field of view; Upon receiving an alarm signal, an alarm will be processed. The three-dimensional alarm zone is determined based on the ground alarm zone configuration of the industrial machinery and the configuration of the fence posts. The human target's position information is the three-dimensional position information of the human target determined by target detection of the three-dimensional point cloud image of the work site in the current frame. The three-dimensional point cloud image is a combination of two-dimensional image and depth information determined by the image data acquisition device at the work site. The alarm signal is generated after determining the positioning result of the corresponding human target inside and outside the fence based on the three-dimensional position information of the human target and the three-dimensional alarm zone, and determining the alarm trigger based on the positioning result.

2. The method according to claim 1, characterized in that, The method for determining the location result of the corresponding human target inside and outside the fence includes: determining the effective three-dimensional point of the human target based on the three-dimensional position information, and determining the location result of the corresponding human target inside and outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area.

3. The method according to claim 2, characterized in that, The method for obtaining the three-dimensional position information includes: using a pre-trained human body tracking model to perform target detection and tracking on the three-dimensional point cloud image, and outputting a corresponding segmentation mask and trajectory information for each human body target detected in the three-dimensional point cloud image; The determination of the valid 3D points is based on the trajectory information and the segmentation mask.

4. The method according to claim 3, characterized in that, The human body tracking model includes a multi-task detection sub-model and a tracking sub-model; In the multi-task detection sub-model, the three-dimensional point cloud image is subjected to target detection processing. For each detected human target, a segmentation mask is determined, and its corresponding human target box and at least one specified part target box are predicted. Based on the human body bounding box and at least one specified body part bounding box, REID features for personnel tracking are determined; In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

5. The method according to claim 4, characterized in that, In the multi-task detection sub-model, the target category is predicted through an attention mechanism, and a corresponding convolutional kernel is generated for each target. The first target is segmented based on the convolutional kernel corresponding to the first target classified as human, and a segmentation mask corresponding to the first target is obtained. Based on the convolutional kernel corresponding to the first target, the human target box and at least one specified part target box corresponding to the first target are predicted by predicting the target box corner heatmap. The segmentation mask corresponding to the first target is multiplied with the image features of the current frame image, and the REID features of the first target are obtained after post-processing.

6. The method according to claim 4, characterized in that, The similarity calculation method between the first target and the first historical trajectory includes: Using the features of the first target as the center, calculate the first matching probability between the first target and all historical trajectories; Using the features of the first trajectory as the center, calculate the second matching probability between the first trajectory and each human target in the current frame; The average of the first matching probability and the second matching probability is used as the similarity between the first target and the first historical trajectory.

7. The method according to claim 5, characterized in that, The prediction obtains the human body bounding box and at least one specified body part bounding box corresponding to the first target, including: Based on the convolution kernel corresponding to the first target, the probability of each target point being a corner point of any target box is predicted. The probability is used as the weight of the corresponding target point, and the weighted average of the coordinates of each target point is calculated. The calculation result is used as the corner point of any target box. The any target box is one of the human body target box corresponding to the first target and at least one target box of a specified part.

8. The method according to claim 3 or 4, characterized in that, After performing target detection on the three-dimensional point cloud image, the human target bounding box corresponding to each human target is further determined; After determining that the first human target is located within the fence or within a set distance from the fence in the current frame, and before determining to trigger an alarm based on the location result, the method further includes: The human target bounding box corresponding to the first human target and the image in which it is located are input into the pre-trained false detection removal model; in the false detection removal model, feature extraction is performed based on the input image and the bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection base library to determine whether the current frame of the first human target is a false detection target and its confidence level. When determining whether an alarm is triggered, further analysis is performed based on the first result and its confidence level. The false detection base database stores the registered false detection target information, including: the target bounding box information of the registered false detection target input by the user, and the embedding vector of the registered false detection target obtained by processing the target bounding box of the registered false detection target and the image where the target bounding box is located using the false detection model.

9. The method according to claim 8, characterized in that, After performing target detection on the three-dimensional point cloud image, at least one designated part of the target box corresponding to each human body target is further determined; In the false detection removal model, feature extraction is further performed based on the at least one specified part target box, and binary classification is performed based on the feature extraction results to predict whether the first human target in the current frame is indeed a human target; when determining to trigger an alarm, it is further based on the prediction results and their confidence levels.

10. The method according to claim 9, characterized in that, The false detection removal model includes a backbone network for extracting image features, a first branch network for binary classification based on the extracted image features, and a second branch network for retrieval based on the extracted image features and the registered false detection target information. When training the false detection removal model, the first branch network performs binary classification on the extracted image features and uses a first loss function to minimize the difference between the predicted probability distribution and the true distribution; the second branch network maps the extracted image features to an embedding space and uses a second loss function to optimize the embedding space, so that similar targets are clustered and dissimilar targets are separated. The training data for the false detection removal model includes two groups of training data. The first group of training data is processed using the backbone network and the first branch network to calculate the value of the first loss function. The second group of training data is processed using the backbone network and the second branch network to calculate the value of the second loss function. The weighted sum of the values ​​of the first and second loss functions is then used as the value of the joint loss function to update the model parameters.

11. The method according to claim 1, characterized in that, The method for determining the three-dimensional alarm zone of the industrial machinery based on the ground alarm zone configuration and fence post configuration includes: Based on the ground alarm zone configuration, a regular polygonal region of the ground in the world coordinate system is determined. Based on the configuration of the fence posts, determine the projection point of the straight line where the fence posts are located to the ground, and update the vertices of the polygonal regular area on the ground based on the projection point; A ground plane is obtained by fitting the vertices of the regular polygonal region of the ground, and all the vertices are projected onto the ground plane to generate the edges of the ground plane. The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

12. The method according to claim 11, characterized in that, Before acquiring and displaying the three-dimensional alarm area, the method further includes: receiving the ground alarm area configuration and / or the fence post configuration input by the user through the configuration interface; or, The configuration of the ground alarm zone and / or the configuration of the fence posts are determined based on the data collected by the image data acquisition device.

13. The method according to claim 11, characterized in that, When generating the side of the fence, retain the three-dimensional points of the fence posts when determining the straight line where the fence posts are located based on the fence post configuration; if the fitted fence post is located outside the three-dimensional alarm area, then the fence post is ignored.

14. The method according to claim 2, characterized in that, The determination of the positioning result of the corresponding human target inside and outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area includes: The relative positional relationship between each effective three-dimensional point and the side of the three-dimensional alarm area is determined point by point, and the positioning result of the corresponding human target inside and outside the fence is determined based on the judgment results of all effective three-dimensional points.

15. The method according to claim 9, characterized in that, The step of determining the alarm trigger based on the location result includes: Based on the positioning results corresponding to multiple frames and the corresponding confidence scores, the confidence scores of the human target bounding box and at least one specified part target bounding box, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information, it is determined whether to trigger an alarm.

16. The method according to claim 1, characterized in that, The method further includes: if there is a closed blind zone at the work site, acquiring and displaying the number of human targets located in the closed blind zone; and when the count result is not 0, triggering an alarm.

17. A monitoring system for the operational safety of industrial machinery, characterized in that, include: Image data acquisition equipment, processor, and display; The image data acquisition device is used to collect data at the work site; The processor is configured to determine the three-dimensional alarm area of ​​the industrial machinery within the imaging field of view of the image data acquisition device based on the ground alarm area configuration and fence post configuration of the industrial machinery; it is also configured to acquire a three-dimensional point cloud image of the work site in the current frame, and acquire the three-dimensional position information of the human target detected in the three-dimensional point cloud image in the current frame; it is further configured to determine the positioning result of the corresponding human target inside and outside the fence in the current frame based on the three-dimensional position information of the human target and the three-dimensional alarm area, and determine whether to trigger an alarm based on the positioning result, and notify the display when it is determined that an alarm is triggered; wherein, the three-dimensional point cloud image is a combination of two-dimensional image and depth information determined based on the acquired data; The display is used to acquire and display the three-dimensional alarm area of ​​the industrial machinery, and also to acquire and display the position information of human targets appearing within the imaging field of view, and to perform alarm processing after receiving an alarm signal.

18. The system according to claim 17, characterized in that, In the processor, determining the positioning result of the corresponding human target inside or outside the fence includes: determining the effective three-dimensional point of the human target based on the three-dimensional position information, and determining the positioning result of the corresponding human target inside or outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area.

19. The system according to claim 18, characterized in that, In the processor, the method of acquiring the three-dimensional position information includes: using a pre-trained human body tracking model to perform target detection and tracking on the three-dimensional point cloud image, and outputting a corresponding segmentation mask and trajectory information for each human body target detected in the three-dimensional point cloud image; In the processor, the effective three-dimensional points are determined based on the trajectory information and the segmentation mask.

20. The system according to claim 19, characterized in that, In the processor, the human body tracking model includes a multi-task detection sub-model and a tracking sub-model; In the multi-task detection sub-model, the three-dimensional point cloud image is subjected to target detection processing. For each detected human target, a segmentation mask is determined, and its corresponding human target box and at least one specified part target box are predicted. Based on the human body bounding box and at least one specified body part bounding box, REID features for personnel tracking are determined; In the tracking sub-model, the human bounding box, at least one specified body part bounding box, and REID features corresponding to the first target output by the multi-task detection sub-model are received, the features are updated, and the similarity between the first target and each historical trajectory is calculated based on the updated features and the relevant features of the historical trajectory. Based on the similarity calculation results, the trajectory information of the first target, the updated human bounding box, and at least one specified body part bounding box are determined.

21. The system according to claim 19 or 20, characterized in that, In the processor, after performing target detection on the 3D point cloud image, it is further used to determine the human target bounding box corresponding to each human target; it is also used to input the human target bounding box corresponding to the first human target and the image in which it is located into a pre-trained false detection removal model after determining that the current frame of the first human target is located within the fence or within a set distance from the fence; in the false detection removal model, feature extraction is performed based on the input image and target bounding box, and an embedding vector is generated. The embedding vector is compared with the registered false detection target information stored in the false detection base library to determine whether the current frame of the first human target is a false detection target and its confidence level. Further, based on the first result and its confidence level, determine whether to trigger an alarm; The false detection base database stores the registered false detection target information, including: the target bounding box information of the registered false detection target input by the user, and the embedding vector of the registered false detection target obtained by processing the target bounding box of the registered false detection target and the image where the target bounding box is located using the false detection model.

22. The system according to claim 21, characterized in that, In the processor, after performing target detection on the 3D point cloud image, it is further used to determine at least one designated part target box corresponding to each human target; it is also used in the false detection removal model to further perform feature extraction based on the at least one designated part target box, and to perform binary classification processing based on the feature extraction result to predict whether the first human target in the current frame is indeed a human target; and to further determine whether to trigger an alarm based on the prediction result and its confidence level.

23. The system according to claim 16, characterized in that, In the processor, determining the three-dimensional alarm zone of the industrial machinery based on the ground alarm zone configuration and the fence post configuration includes: Based on the ground alarm zone configuration, a regular polygonal region of the ground in the world coordinate system is determined. Based on the configuration of the fence posts, determine the projection point of the straight line where the fence posts are located to the ground, and update the vertices of the polygonal regular area on the ground based on the projection point; A ground plane is obtained by fitting the vertices of the regular polygonal region of the ground, and all the vertices are projected onto the ground plane to generate the edges of the ground plane. The fence side is generated based on each edge of the ground plane, and the three-dimensional alarm area is obtained based on the ground plane and the fence side.

24. The system according to claim 23, characterized in that, The display is further configured to receive the ground alarm zone configuration and / or the fence post configuration input by the user through the configuration interface, and forward them to the processor; or, The processor is further configured to determine the ground alarm zone configuration and / or the fence post configuration based on the data acquired by the image data acquisition device.

25. The system according to claim 18, characterized in that, In the processor, determining the positioning result of the corresponding human target inside and outside the fence based on the relative positional relationship between the effective three-dimensional point and the three-dimensional alarm area includes: The relative positional relationship between each effective three-dimensional point and the side of the three-dimensional alarm area is determined point by point, and the relative positional relationship between the corresponding human target and the three-dimensional alarm area is determined based on the determination results of all effective three-dimensional points.

26. The system according to claim 22, characterized in that, In the processor, determining whether to trigger an alarm based on the positioning result includes: Based on the positioning results corresponding to multiple frames and the corresponding confidence scores, the confidence scores of the human target bounding box and at least one specified part target bounding box, the segmentation confidence score of the segmentation mask, the area of ​​the segmentation mask, the trajectory information of the human target, the trajectory confidence score, the continuous tracking duration, the classification confidence score of the human target, the number of classifications, the false detection confidence score, and / or the target height information, it is determined whether to trigger an alarm.

27. The system according to claim 17, characterized in that, When there is a closed blind zone at the work site, the processor is further used to count the human targets located in the closed blind zone and send the counting result to the display; when the counting result is not 0, an alarm is triggered and the display is notified. The display is used to acquire and display the number of human targets located in the closed blind area when there is a closed blind area at the work site; and to perform an alarm process when it receives a notification that the result of the processor is not 0.

28. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it can realize the monitoring method for industrial machinery operation safety as described in any one of claims 1 to 16.

29. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the monitoring method for industrial machinery operation safety as described in any one of claims 1 to 16.

Images