A safe passage blockage detection method, device, equipment and storage medium

Abstract

The application discloses a safety passage blockage detection method, device and equipment and a storage medium, the method filters out non-blocking objects and pedestrians by using an image target detection technology, then judges whether a moving object is blocked by using an image tracking algorithm, finally, a metric learning algorithm is used to complete similarity comparison between a final blocked picture and a non-blocked picture, and whether the safety passage is blocked is determined, the method has the characteristics of non-contact and indirect detection, and has the characteristics of strong adaptability and high generalization capability. The method can be seamlessly deployed in different factories.

Description

Technical Field

[0001] This invention relates to the field of factory safety management technology, and in particular to a method, apparatus, equipment and storage medium for detecting obstructions in safety passages suitable for use in factory environments. Background Technology

[0002] Recently, safety accidents have occurred frequently in factory settings, including but not limited to overcrowding caused by blocked safety exits and situations where fires prevent escape through safety exits. Therefore, it is clear that unobstructed safety exits are a crucial prerequisite for crowd evacuation and fire escape, and a vital part of safe escape. Thus, timely alarms and prompt clearing of blockages in safety exits are of paramount importance.

[0003] Various methods have been tried to detect obstructions in safety passages. Initially, the main approach involved manually reviewing surveillance videos and monitoring in real time. However, this method required personnel to maintain high concentration for extended periods, making it impractical for large-scale factory deployments. It was time-consuming, labor-intensive, and costly, and was therefore largely abandoned. With the development of image detection technology, a solution utilizing object detection technology for obstruction detection has emerged. This solution detects obstructions by identifying the presence of objects within the passageway.

[0004] However, due to the complex factory environment, the variety of obstructing objects, and the lack of fixed category restrictions, traditional target detection methods for safety passage obstruction detection often fail to provide comprehensive coverage. For example, movable objects such as pedestrians and moving vehicles, which are not obstructions, are frequently misidentified when detected.

[0005] Therefore, how to provide a method for detecting safety passage blockages in a factory environment, which can accurately detect passage blockages in different environments within a factory, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method, apparatus, device, and storage medium for detecting obstructions in safe passages to overcome or at least partially solve the above problems. It can effectively detect obstructions in passages under different environments in a factory, and also has scalability; the feature extraction network can be customized according to different factory environments, exhibiting strong generalization ability.

[0007] This invention provides the following solution:

[0008] A method for detecting obstruction in a safety exit includes:

[0009] Acquire a test image, wherein the test image is an image of a target area acquired by an image capturing device;

[0010] The trained object detection network is used to detect and determine if the image contains a target object; the target object includes movable objects other than people.

[0011] Multiple frames of sampled images containing the target object are continuously acquired, and the target detection network is used to detect the multiple frames of sampled images to obtain multiple target detection boxes;

[0012] The position of each target detection box is predicted using Kalman filtering, and the similarity matching is performed using the Hungarian algorithm to obtain the movement trajectory of the target object;

[0013] Determine whether the target object has moved based on the movement trajectory;

[0014] Once it is determined that the target object is in a moving state, the target object in the image to be detected is blacked out using the trained metric learning base network to obtain the target image;

[0015] The target image is compared with images in the non-blocking database using a metric learning method. The similarity between the two images is calculated using cosine distance. Based on the similarity calculation results, it is determined whether there is a blockage.

[0016] Preferably, the target detection network includes a YOLOv5 target detection network; the base network of the YOLOv5 target detection network uses CSPDarknet53 to extract features from different layers of the target image; the fusion network of the YOLOv5 target detection network uses an SPP module combined with FPN and PAN structures for feature fusion; and the loss function of the YOLOv5 target detection network uses CI OU_Loss.

[0017] Preferably: The characteristics of factory lighting are learned through GAN data generation technology, and the collected samples are expanded for different lighting and environmental conditions to simulate and generate images under different lighting conditions;

[0018] Partially occluded and blurred samples are added as difficult samples;

[0019] Using image annotation work, the movable object class samples in the VOC2007 dataset are labeled according to the labeling format to obtain a detection sample dataset; the sample dataset is used for training the target detection network.

[0020] Preferably: 30 frames of sampled images containing the target object are continuously acquired; and the target detection network is used to detect each of the 30 sampled images to obtain 30 target detection boxes;

[0021] Determining whether the target object has moved based on the movement trajectory includes:

[0022] Match the first frame result with the bounding box of each subsequent frame result until the last frame result;

[0023] If the center of gravity of the Bbox shifts within 20 pixels, the target object is determined to be stationary.

[0024] If the center of gravity shifts within 30 frames and follows a motion trajectory, the target object is determined to be in a moving state.

[0025] Preferably, the metric learning base network includes a network architecture consisting of a ResNet network combined with an ArcFace Loss function.

[0026] Preferably: the metric learning base network is used to extract features from the images in the gallery library, and the vector after the adaptive average pooling layer is selected as the feature vector emb_g and stored in the gallery library.

[0027] Preferably, the similarity calculation result is normalized to a percentage system. When the similarity exceeds 90%, the channel is determined to be in a non-blocked state; when the similarity is less than 90%, the safe channel is determined to be blocked.

[0028] A safety passage blockage detection device, the device comprising:

[0029] The image acquisition unit is used to acquire an image to be tested, wherein the image to be tested is an image of a target area acquired by an image capturing device;

[0030] The target object determination unit is used to detect and determine whether the image to be tested contains a target object using a trained target detection network; the target object includes movable objects other than people.

[0031] The target detection box acquisition unit is used to continuously acquire multiple frames of sampled images containing the target object, and use the target detection network to detect the multiple frames of sampled images to obtain multiple target detection boxes;

[0032] The movement trajectory acquisition unit is used to predict the position of each target detection box using Kalman filtering and to perform similarity matching using the Hungarian algorithm to obtain the movement trajectory of the target object;

[0033] A movement determination unit is used to determine whether the target object has moved at this time based on the movement trajectory.

[0034] The target image acquisition unit is used to determine that the target object is in a moving state, and to use the trained metric learning base network to black out the target object in the image to be detected to obtain the target image;

[0035] The congestion determination unit is used to compare the features of the target image with images in the non-congestion database using a metric learning method, calculate the similarity between the two images using cosine distance, and determine whether there is a congestion based on the similarity calculation result.

[0036] A detection device, the device comprising a processor and a memory:

[0037] The memory is used to store program code and transmit the program code to the processor;

[0038] The processor is used to execute the above-described secure channel blockage detection method according to the instructions in the program code.

[0039] A computer-readable storage medium for storing program code for executing the above-described secure channel blockage detection method.

[0040] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0041] This application provides a method, apparatus, device, and storage medium for detecting safety passage blockages. It utilizes image target detection technology to filter out non-blocking objects and pedestrians, then employs an image tracking algorithm to determine whether moving objects constitute a blockage. Finally, a metric learning algorithm is used to compare the similarity between the blocked and non-blocked images to determine if the safety passage is blocked. This method is characterized by non-contact, indirect detection, and also exhibits strong adaptability and high generalization ability. It can be seamlessly deployed in different factories.

[0042] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0044] Figure 1 This is a flowchart of a safety passage blockage detection method provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the target detection network provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the target detection results of blockage in a factory workshop provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a daytime lighting image under a camera in a factory, provided in an embodiment of the present invention.

[0048] Figure 5 This is a schematic diagram of a nighttime lighting image under a camera in a factory, provided in an embodiment of the present invention.

[0049] Figure 6 This is the original congested input image from a camera in a factory, provided in an embodiment of the present invention.

[0050] Figure 7 This is the input image obtained by excluding non-blocking interference images from a camera in a factory, as provided in this embodiment of the invention.

[0051] Figure 8 This is a schematic diagram of a safety passage blockage detection device provided in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of a safety passage blockage detection device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0054] See Figure 1 This invention provides a method for detecting obstructions in a safe passage, such as... Figure 1 As shown, the method may include:

[0055] S101: Acquire the image to be tested, which is an image of the target area acquired by an image capturing device; in specific implementation, images can be acquired by cameras deployed inside the factory area.

[0056] S102: The trained object detection network is used to detect and determine the presence of target objects in the image to be tested; the target objects include movable objects (vehicles, etc.) other than people; specifically, the object detection network includes the YOLOv5 object detection network; the base network of the YOLOv5 object detection network uses CSPDarknet53 to extract features from different layers of the target image; the fusion network of the YOLOv5 object detection network uses an SPP module combined with FPN and PAN structures for feature fusion; the loss function of the YOLOv5 object detection network is CI OU_Loss.

[0057] To eliminate the impact of different lighting variations on image similarity judgment, this application embodiment can also provide learning the factory lighting characteristics through GAN data generation technology, expanding the collected samples for different lighting and environmental conditions, and simulating the generation of images under different lighting conditions;

[0058] Partially occluded and blurred samples are added as difficult samples;

[0059] Using image annotation work, the movable object class samples in the VOC2007 dataset are labeled according to the labeling format to obtain a detection sample dataset; the sample dataset is used for training the target detection network.

[0060] S103: Continuously acquire multiple frames of sampled images containing the target object, and use the target detection network to detect the multiple frames of sampled images to obtain multiple target detection boxes;

[0061] S104: Kalman filtering is used to predict the position of each target detection box, and Hungarian algorithm is used for similarity matching to obtain the movement trajectory of the target object; specifically, 30 consecutive sampled images containing the target object are acquired; and the target detection network is used to detect the 30 sampled images to obtain 30 target detection boxes;

[0062] Determining whether the target object has moved based on the movement trajectory includes:

[0063] Match the first frame result with the bounding box of each subsequent frame result until the last frame result;

[0064] If the center of gravity of the Bbox shifts within 20 pixels, the target object is determined to be stationary.

[0065] If the center of gravity shifts within 30 frames and follows a motion trajectory, the target object is determined to be in a moving state.

[0066] S105: Determine whether the target object has moved at this time based on the movement trajectory;

[0067] S106: Determine that the target object is in a moving state, and use the trained metric learning base network to black out the target object in the image to be detected to obtain the target image; specifically, the metric learning base network includes a network architecture of ResNet network combined with ArcFace Loss function. Use the metric learning base network to extract features from images in the gallery library, and select the vector after adaptive average pooling layer as the feature vector emb_g and store it in the gallery library.

[0068] S107: Using metric learning, the target image is compared with images in the non-blocking database in terms of features. The similarity between the two images is calculated using cosine distance. Based on the similarity calculation result, it is determined whether there is a blockage. Specifically, the similarity calculation result is normalized to a percentage. When the similarity exceeds 90%, the channel is determined to be in a non-blocking state; when the similarity is less than 90%, it is determined to be a blocked state in the safe channel.

[0069] The safety passage blockage detection method provided in this application utilizes image detection technology to detect and alarm when passages are blocked in a factory environment. Specifically, it uses cameras deployed within the factory area to capture images, detects non-blocking objects (movable objects) such as pedestrians and vehicles in the images using the YOLOv5 scheme, then creates images of unblocked passages, extracts image features using a ResNet network, and compares these images using a metric learning method. Based on the similarity of the comparison results, the similarity is inverted to determine whether a passage blockage exists and triggers an alarm. This method is applicable to passage blockage detection under different lighting conditions and environments. This scheme can effectively detect passage blockage problems in different factory environments and has scalability; the feature extraction network can be customized according to different factory environments, demonstrating strong generalization ability.

[0070] This method employs a YOLOv5 object detection network and a ResNet feature extraction network for metric learning, and is generally completed in two steps:

[0071] The first step is to construct an object detection framework. Non-blocking objects in the channel, such as pedestrians and moving vehicles, are considered as non-blocking objects. A dataset for non-blocking object detection is established. Then, a YOLOv5 object detection network is constructed. The base network of the detection framework uses CSPDarknet53 to extract features from different layers of the target image. The fusion network uses the SPP module combined with FPN and PAN structures for feature fusion. Finally, the loss function is CI OU_Loss. The detection network is trained on the non-blocking object monitoring dataset to form the basic model for object detection. Detection is performed within a safe channel. If a corresponding target is detected, the target's bounding box information is saved for subsequent use.

[0072] The second step involves constructing a non-blocking gallery library from different cameras, creating an image dataset of factory safety passages. Then, a metric learning network for feature extraction is trained using a ResNet network in conjunction with the ArcFace Loss function. Non-blocking objects in the images to be detected in the first step are blacked out. The metric learning method is then used to compare features with images in the non-blocking gallery, and the similarity between the two images is calculated using cosine distance. Based on the similarity calculation results, it is determined whether there is a blockage. When the similarity is less than 90%, the safety passage is determined to be blocked.

[0073] However, in actual factory safety passages, there are different lighting variations, and vehicles both stop and move within the passageways. Stationary vehicles constitute a blockage, but moving vehicles do not. To address these two situations, the following two solutions are available.

[0074] To address the varying lighting conditions in factories, which typically involve both working and non-working hours, images under different lighting conditions can be added to the non-blocking metric learning library as a supplement to the gallery. By comparing different gallery images at different time points, the issue of image similarity differences caused by varying lighting conditions can be resolved.

[0075] For situations where moving vehicles do not obstruct the safety passage, but stationary vehicles do, this method involves detecting a vehicle in step one and then tracking the vehicle for 30 frames. If there is no change in the vehicle's trajectory within 30 frames, it is considered a blockage; if there is a change in the vehicle's trajectory within 30 frames, it is considered a non-blockage.

[0076] Therefore, this invention uses target detection technology and metric learning technology as the main image detection methods, combined with a trajectory tracking scheme to detect obstructions in safety passages. This method is non-contact, unaffected by factors such as lighting environment, and can detect various types of obstructions without category limitations. In addition, this method has strong generalization ability and can be quickly deployed in different factories.

[0077] This invention aims to use image detection technology to detect and provide real-time alarms for blocked safety passages in factories.

[0078] The specific solution involves using cameras deployed inside the factory area to capture images, detecting non-blocking objects such as pedestrians and vehicles in the images, and using the YOLOv5 algorithm. When a vehicle is detected, a Kalman filter is used to predict the position of each vehicle detection box, and then a Hungarian algorithm is used for similarity matching to obtain the final movement trajectory. The trajectory is then used to determine whether there is a blockage.

[0079] When no vehicle is detected, an image of an unblocked gallery is created. A safety gallery feature extraction model is trained using a ResNet network in conjunction with ArcFace Loss. This model is used to extract features from the image under test. Then, the cosine similarity between the image under test and the corresponding unblocked gallery image from the same camera at the same time is calculated. Based on the similarity results, it is determined whether there is a gallery blockage. If a safety gallery blockage is detected, an alarm is triggered.

[0080] To achieve the objective of this invention—to detect obstructions in safety passages and issue real-time alarms using image detection technology—the following steps are required:

[0081] Step 1: Acquire images using cameras within the factory area. Detect non-blocking pedestrians and vehicles within the images and return bounding boxes (bboxes). Use images acquired from factory cameras as training samples. Label pedestrians and vehicles, which are more easily misidentified, and then use YOLOv5 to detect them. Finally, obtain the detection results.

[0082] Specifically, a non-blocking object detection dataset is established, and then a YOLOv5 object detection network is constructed. The base network of the detection framework uses CSPDarknet53 to extract features from different layers of the target image. The fusion network uses the SPP module in combination with FPN and PAN structures for feature fusion. Finally, the loss function is CI OU_Loss. The detection network is trained on the non-blocking object monitoring dataset to train the basic model for object detection. Detection is performed within a safe passage. If a corresponding target is detected, the target's bounding box information is saved for subsequent use.

[0083] Step 101: First, it is necessary to establish a target detection dataset for non-blocking elements in the factory. Pedestrians and vehicles in motion in the factory area are considered non-blocking factors. Therefore, this method collects 2,000 samples containing various types of vehicles and pedestrians in the factory area as the training set. Since some factory areas have security and confidentiality requirements, a portion is actually collected and a portion is generated using the GAN scheme as samples.

[0084] Step 102: Data augmentation is performed on the vehicle and pedestrian target detection dataset generated in Step 101. Due to the complex environment and lighting conditions in the factory area, GAN data generation technology is used to learn the factory lighting characteristics to expand the collected samples for different lighting and environmental conditions, simulating the generation of images under different lighting conditions. At the same time, some occluded and blurred samples are added as difficult samples. After the above work is completed, image annotation is used to label the pedestrian and vehicle samples according to the labeling format of the VOC2007 dataset. At this point, the target detection dataset for safety passage blockage is completed.

[0085] Step 103: Use the dataset from Step 102 to begin training the object detection framework. The framework structure diagram is as follows: Figure 2 As shown, the main components consist of a feature extraction module, a feature separation and fusion module, and a loss function. The feature extraction network serves as the training backbone; this network is a CSPDarknet53 network constructed by adding a CSP module to the Darknet53 model for feature extraction. Then, the SPP module is used to separate features at different scales. Finally, FPN+PAN is used to complete the feature fusion, ensuring that the network can extract strong semantic features while also conveying strong localization features from the bottom up through a pyramid model. This allows for multi-dimensional and multi-level fusion of feature information. After obtaining the features, it is necessary to determine whether to use a classification loss function or a regression loss function. In this method, the cross-entropy loss function is used for classification, and the CIOU_LOS function is used for regression of the bounding box. CIOU_LOS addresses the aforementioned issues by adding hyperparameters, thus overcoming their influence. The mathematical expression for its implementation is shown in Equation 1.

[0086]

[0087] Where v is the aspect ratio penalty hyperparameter, defined by Equation 2:

[0088]

[0089] Step 104: Using the object detection network constructed in Step 103, train the target detection sample dataset of safety passage blockages from Step 102. The training hardware server is configured with two Tesla T4 graphics cards, a Linux 18.04 system, 64GB of memory, and an Intel Core i9 CPU. Training is performed using 256 images per batch for a total of 120 epochs. After the loss decays to a stable trend, the accuracy in the VAL test reaches 98.86%, the recall reaches 99.2%, and the map reaches 78%, and no further increases are observed. At this point, training is stopped to obtain the optimal model for subsequent predictions.

[0090] Step 105 involves forward prediction of images captured by cameras in the factory safety passage. Using OpenCV decoding tools, the video stream is decoded frame by frame into images. The factory cameras operate at 25 frames per second. This method employs a detection interval of 5 frames to achieve a balance between accuracy and detection efficiency. Before detection, the quality of the images and their content needs to be assessed to ensure that the decoded images are free of static frames, black areas, etc. Next, image sharpness is judged, primarily to avoid image problems caused by motion blur. Only after all the above checks are passed can target detection prediction be performed. The optimal safety passage blockage model trained in step 104 is used for target detection. When pedestrians or vehicles are detected, bounding boxes are used to mark the detection results for subsequent tracking and judgment. The detection process is as follows... Figure 3 As shown, the method demonstrates the results, using a factory workshop as an example.

[0091] Step Two: When a vehicle is detected, 30 consecutive frames are captured. A Kalman filter is used to predict the position of each vehicle detection box, and then a Hungarian algorithm is used for similarity matching to obtain the final movement trajectory. Based on the trajectory results, it is determined whether the vehicle has moved, thus determining whether the passage is blocked. Specifically, when a vehicle (forklift, transport vehicle, etc.) is detected, a Kalman filter is used to predict the position of each vehicle detection box, and then a Hungarian algorithm is used for similarity matching to obtain the final vehicle movement trajectory. If the trajectory does not change within 30 frames, it is considered blocked; if the trajectory changes within 30 frames, it is considered not blocked.

[0092] In step two, when a vehicle is detected in the result label from step 104, 30 consecutive image frames are acquired. A Kalman filter is used to predict the position of each vehicle detection box, and then a Hungarian algorithm is used for similarity matching to obtain the final movement trajectory. Based on the trajectory result, it is determined whether the vehicle has moved, thereby determining whether the passage is blocked. The specific implementation steps of step two are as follows:

[0093] Step 201: After a vehicle appears in the bounding box obtained in step 104, this result is used as the initialization result for `detect`. Kalman filtering is used to initialize this result as tracking trajectories (tracks). Then, the results of the next 30 consecutive frames are treated as a continuous sequence L. Each detection result labeled as a vehicle in sequence L is denoted as M. x Using Kalman filtering to predict subsequent M x The state is estimated by using the mean and covariance. x The relationship with the initial tracks is represented by Equation 3.

[0094] M x =[x,y,r,h,x ′ ,y ′ ,r ′ ,h ′ (Equation 3)

[0095] Where x and y represent the centroid coordinates of the bounding box, r represents the aspect ratio, and h represents the height.

[0096] Step 202: After completing the initialization and Kalman filter prediction in step 201, M... x The result is used as a new detect for prediction, and then the Hungarian algorithm is used for matching to determine the matching pair (track, detection) at this moment. This method uses motion features and appearance features to perform cost matrix operations to complete target matching. The motion features use Mahalanobis distance to predict the distance between the state of the Kalman filter in step 201 and the newly obtained detections, as expressed in Equation 4.

[0097] D (i,j) =(D i —M i ) T S i ―1 (D i —M i (Equation 4)

[0098] Among them, M i S is the mean. i Since the covariance is used, the Mahalanobis distance is used to estimate the uncertainty between the two states by calculating the covariance between the means of the Kalman filter tracking positions. By setting a threshold for the Mahalanobis distance, associated targets are identified and unassociated targets are excluded. The threshold range set in this method is greater than 0.9, which can reasonably exclude unassociated targets.

[0099] Appearance features are processed using a Mob Il Enet network. This network extracts 104 results from the targets in the detectors as input and stores the appearance features. When the next 30 consecutive vehicle detectors are entered, appearance matching is performed with the features of each detector in the initial test track. The Euclidean distance with each detector is calculated. When the appearance similarity is greater than the appearance similarity threshold of 0.65, the appearance of the detector is determined and added to the initial test track.

[0100] Step 203: For the tracks that were successfully matched in step 202, repeat step 202 to match the subsequent 30 consecutive results.

[0101] Step 204: If a detect that was not matched in step 203 fails to find a corresponding track more than 5 times, initialize it as a new track and discard the remaining detects.

[0102] Step 205: At this point, 30 consecutive frames of vehicle trajectory routes are obtained. After determining the trajectory routes, the first frame result is matched with the subsequent frame results using bounding boxes until the last frame result. If the center of gravity of the bounding box shifts within 20 pixels, it can be determined that the vehicle is stationary and falls within the safe passage blockage area. If the center of gravity shifts in the 30 frames and follows a moving trajectory, it can be determined that the vehicle is a moving vehicle in the safe passage and does not constitute a safe passage blockage.

[0103] Step 3: Utilize a ResNet network in conjunction with ArcFace Loss to establish a feature extraction network for the safe passage region. Train the model using factory images and compare the similarity of the test image with unblocked images in the corresponding camera's gallery to obtain the similarity results. Specifically, if the image was determined to be unblocked in Step 2, train a metric learning network for feature extraction using ResNet and ArcFace Loss. Blacken the unblocked objects in the image from Step 1, then compare the features with images in the unblocked database using metric learning methods. Calculate the similarity between the two images using cosine distance. Based on the similarity calculation results, determine whether there is a blockage. When the similarity is below 90%, it is determined to be a blocked safe passage.

[0104] In step three, a secure channel feature extraction network is established using a ResNet network in conjunction with ArcFace Loss. The model is trained using factory images, and the similarity is compared between the image to be tested and the unblocked images in the corresponding camera's gallery to obtain the similarity results. The specific implementation steps of step three are as follows:

[0105] Step 301: Establish a metric learning dataset. Since this method is mainly applied in factory scenarios, factory samples are used as the dataset. A ResNet network is established with the following parameters: depth set to 50, dropout_rate set to 0.5, mode selected as i_se, output embedding_size set to 1024, input size set to 224*224, PIXEL_MEAN set to [0.485, 0.456, 0.406], PIXEL_STD set to [0.229, 0.224, 0.225], and metric_loss selected as ArcFace Loss. ArcFace Loss is an improved version of softmax loss. It normalizes the output X of the penultimate layer of the network, then normalizes the weight value of the last layer, and multiplies it by a constant factor S. Then, from a geometric perspective, θ can be found to be the segmentation curve of the class. Adding the interval m on this basis, we get the complete ArcFace Loss, as shown in Equation 5. A feature extraction model for a factory environment is trained using the dataset and algorithm samples.

[0106]

[0107] Step 302: Establish a non-blocking gallery library for each camera. Due to the special nature of the factory environment, there are generally two different situations: daytime lighting and nighttime lighting. Therefore, two types of image features are added to the gallery library. Taking an image captured by a camera in the factory area as an example, the daytime and nighttime lighting conditions are as follows: Figure 4 , Figure 5 As shown, using the feature extraction model obtained in step 301, features are extracted from the images in the library. The vector after the AdaptiveAvgPool layer is selected as the feature vector emb_g and stored in the vector library.

[0108] Step 303: Based on the target detection results in Step 104 and the trajectory tracking results in Step 205, fill the non-blocking objects in the image with RGB values ​​of (255, 255, 255), and simultaneously fill the corresponding positions in the gallery image in Step 302 with (255, 255, 255). Figure 6 , Figure 7For example, according to step 205, the vehicle is determined to be a stopped vehicle and is considered a blockage, while step 104 determines that the pedestrian is not a blockage. Therefore, the pedestrian vehicle detection frame is filled with RGB values ​​(255,255,255) to obtain the final image result to be compared.

[0109] Step 304: Using the feature extraction model obtained in step 301, extract image features from the results obtained in step 303. Then, calculate the similarity between the images in the remaining gallery database using the cosine distance formula. Normalize the results to a percentage. When the similarity exceeds 90%, the channel is in a non-blocked state; conversely, when the similarity is less than 90%, it is judged to be in a blocked state. Figure 6 , Figure 7 The result showed a similarity of 59%, indicating that the safety passage was blocked.

[0110] Step 4: Based on the similarity comparison results, determine whether a fire escape route is blocked and whether an alarm needs to be triggered. Based on the results of Steps 2 and 3, determine the extent of the blockage. When a blockage is detected, transmit the result back to the monitoring system and trigger the audible and visual alarm.

[0111] In step four, based on the similarity comparison results, it is determined whether a fire escape is blocked and whether an alarm needs to be sounded. The specific implementation steps of step four are as follows:

[0112] Step 401, as described in step 105, involves acquiring and judging images every 5 frames. However, this single judgment may lead to false positives. If 9 out of 10 consecutive results show a blockage, the safety passage under the camera is determined to be blocked. The results are then uploaded to the monitoring system, triggering an alarm with sound and light.

[0113] Combining the above four steps, video image detection technology can be used to detect obstructions in safety passages in a factory environment. This method achieves real-time, accurate, and efficient full coverage detection of all safety passages through indirect detection. It combines object detection, object tracking, and metric learning techniques in image algorithms to achieve adaptability not found in single object detection or single metric learning solutions. Furthermore, this method can be used as a standalone solution or as a module in a smart monitoring system to link with audible and visual alarms. It exhibits strong generalization capabilities for both object detection and metric learning, allowing for seamless deployment in new factory environments. By adding categories and sample images, it can also be used to implement safety passage detection solutions in other areas.

[0114] In summary, the safety passage obstruction detection method provided in this application utilizes image target detection technology to filter out non-obstructing objects and pedestrians, then employs an image tracking algorithm to determine whether moving objects constitute an obstruction, and finally uses a metric learning algorithm to compare the similarity between the obstructed and non-obstructed images to determine whether the safety passage is indeed blocked. This method features non-contact, indirect detection, as well as strong adaptability and high generalization ability. It can be seamlessly deployed in different factories.

[0115] See Figure 8 This application embodiment can also provide a safety passage blockage detection device, such as... Figure 9 As shown, the device may include:

[0116] Image acquisition unit 801 is used to acquire a test image, wherein the test image is an image of a target area acquired by an image capturing device;

[0117] The target object determination unit 802 is used to detect and determine the presence of target objects in the image to be tested using a trained target detection network; the target objects include movable objects other than people.

[0118] The target detection box acquisition unit 803 is used to continuously acquire multiple frames of sampled images containing the target object, and use the target detection network to detect the multiple frames of sampled images to obtain multiple target detection boxes;

[0119] The movement trajectory acquisition unit 804 is used to predict the position of each target detection box using Kalman filtering and to perform similarity matching using the Hungarian algorithm to obtain the movement trajectory of the target object;

[0120] The movement determination unit 805 is used to determine whether the target object has moved at this time based on the movement trajectory.

[0121] The target image acquisition unit 806 is used to determine that the target object is in a moving state, and to obtain the target image by blackening the target object in the image to be detected using a trained metric learning base network.

[0122] The congestion determination unit 807 is used to compare the features of the target image with images in the non-congestion database using a metric learning method, calculate the similarity between the two images using cosine distance, and determine whether there is a congestion based on the similarity calculation result.

[0123] This application embodiment can also provide a safety passage blockage detection device, characterized in that the device includes a processor and a memory:

[0124] The memory is used to store program code and transmit the program code to the processor;

[0125] The processor is used to execute the steps of the above-described secure channel blockage detection method according to the instructions in the program code.

[0126] like Figure 9 As shown in the figure, a safe passage blockage detection device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0127] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0128] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the secure channel blockage detection method.

[0129] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0130] Acquire a test image, wherein the test image is an image of a target area acquired by an image capturing device;

[0131] The trained object detection network is used to detect and determine if the image to be tested contains a target object; the target object includes a movable object.

[0132] Multiple frames of sampled images containing the target object are continuously acquired, and the target detection network is used to detect the multiple frames of sampled images to obtain multiple target detection boxes;

[0133] The position of each target detection box is predicted using Kalman filtering, and the similarity matching is performed using the Hungarian algorithm to obtain the movement trajectory of the target object;

[0134] Determine whether the target object has moved based on the movement trajectory;

[0135] Once it is determined that the target object is in a moving state, the target object in the image to be detected is blacked out using the trained metric learning base network to obtain the target image;

[0136] The target image is compared with images in the non-blocking database using a metric learning method. The similarity between the two images is calculated using cosine distance. Based on the similarity calculation results, it is determined whether there is a blockage.

[0137] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0138] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0139] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0140] Of course, it should be noted that, Figure 9 The structure shown does not constitute a limitation on the safety passage blockage detection device in the embodiments of this application. In practical applications, the safety passage blockage detection device may include more than Figure 9 More or fewer components as shown, or combinations of certain components.

[0141] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described security channel blockage detection method.

[0142] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0143] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting obstruction in a safety passage, characterized in that, include: Acquire a test image, wherein the test image is an image of a target area acquired by an image capturing device; The trained object detection network is used to detect and determine the presence of target objects in the test image; the target objects include movable objects other than people; the object detection network includes the YOLOv5 object detection network; the base network of the YOLOv5 object detection network uses CSPDarknet53 to extract features from different layers of the target image; the fusion network of the YOLOv5 object detection network uses an SPP module combined with FPN and PAN structures for feature fusion; the loss function of the YOLOv5 object detection network is CIOU_Loss. Multiple frames of sampled images containing the target object are continuously acquired, and the target detection network is used to detect the multiple frames of sampled images to obtain multiple target detection boxes; The position of each target detection box is predicted using Kalman filtering, and the similarity matching is performed using the Hungarian algorithm to obtain the movement trajectory of the target object; Determine whether the target object has moved based on the movement trajectory; Once it is determined that the target object is in a moving state, the target object in the image to be detected is blacked out using the trained metric learning base network to obtain the target image; The target image is compared with images in the non-blocking database using a metric learning method. The similarity between the two images is calculated using cosine distance. Based on the similarity calculation results, it is determined whether there is a blockage.

2. The safety passage blockage detection method according to claim 1, characterized in that, By learning the characteristics of factory lighting through GAN data generation technology, the collected samples are expanded for different lighting and environmental conditions to simulate and generate images under different lighting conditions. Partially occluded and blurred samples are added as difficult samples; Using image annotation work, the movable object class samples in the VOC2007 dataset are labeled according to the labeling format to obtain a detection sample dataset; the sample dataset is used for training the target detection network.

3. The safety passage blockage detection method according to claim 1, characterized in that, 30 consecutive sampled images containing the target object are acquired; and the target detection network is used to detect each of the 30 sampled images to obtain 30 target detection boxes; Determine whether the target object has moved based on the movement trajectory; include: Match the first frame result with the bounding box of each subsequent frame result until the last frame result; If the center of gravity of the Bbox shifts within 20 pixels, the target object is determined to be stationary. If the center of gravity shifts within 30 frames and follows a motion trajectory, the target object is determined to be in a moving state.

4. The safety passage blockage detection method according to claim 1, characterized in that, The underlying network for metric learning includes a network architecture that combines the ResNet network with the ArcFace Loss function.

5. The safety passage blockage detection method according to claim 4, characterized in that, The metric learning base network is used to extract features from images in the gallery library. The vector after adaptive average pooling is selected as the feature vector emb_g and stored in the vector gallery library.

6. The safety passage blockage detection method according to claim 1, characterized in that, The similarity calculation results are normalized to a percentage system. When the similarity exceeds 90%, the channel is determined to be in a non-blocked state; when the similarity is less than 90%, the safe channel is determined to be blocked.

7. A safety passage blockage detection device, characterized in that, The apparatus for performing the safety passage obstruction detection method according to any one of claims 1-6, the apparatus comprising: The image acquisition unit is used to acquire an image to be tested, wherein the image to be tested is an image of a target area acquired by an image capturing device; The target object determination unit is used to detect and determine whether the image to be tested contains a target object using a trained target detection network; the target object includes movable objects other than people. The target detection box acquisition unit is used to continuously acquire multiple frames of sampled images containing the target object, and use the target detection network to detect the multiple frames of sampled images to obtain multiple target detection boxes; The movement trajectory acquisition unit is used to predict the position of each target detection box using Kalman filtering and to perform similarity matching using the Hungarian algorithm to obtain the movement trajectory of the target object; A movement determination unit is used to determine whether the target object has moved at this time based on the movement trajectory. The target image acquisition unit is used to determine that the target object is in a moving state, and to use the trained metric learning base network to black out the target object in the image to be detected to obtain the target image; The congestion determination unit is used to compare the features of the target image with images in the non-congestion database using a metric learning method, calculate the similarity between the two images using cosine distance, and determine whether there is a congestion based on the similarity calculation result.

8. A testing device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the security passage blockage detection method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the security passage blockage detection method according to any one of claims 1-6.

Images