Stranger loitering detection method, apparatus, device and storage medium

Abstract

This invention discloses a method, apparatus, device, and storage medium for detecting stranger loitering. The stranger loitering detection method provided in this application features high detection accuracy and excellent detection efficiency. Furthermore, this method employs a non-contact solution, enabling contactless, full-coverage, 24 / 7 real-time detection. It also utilizes a standalone deployment method using a server or industrial control computer, making it adaptable to both factories with existing cameras and those without. Therefore, this method is characterized by rapid and convenient deployment. Clearly, this method is highly adaptable to the construction of smart factories under Industry 4.0.

Description

Technical Field

[0001] This invention relates to the field of deep learning image processing technology, and in particular to a method, apparatus, device, and storage medium for detecting stranger loitering. Background Technology

[0002] With the widespread adoption of Industry 4.0, the construction of intelligent factories involving sensitive equipment is drawing ever closer. Since 2016, deep learning image processing technology has comprehensively updated traditional image algorithms in various aspects, significantly improving accuracy, efficiency, and generalization ability compared to traditional methods. Therefore, the need to apply deep learning image processing technology to factory scenarios has emerged. Personnel management and monitoring have always been challenging issues for factories. Given the complexity of the factory environment, how to achieve refined personnel management has become a primary task for intelligent factory inspection.

[0003] There are some traditional solutions in the existing technology to detect strangers loitering in the factory area. The earliest solution was to use human patrols and monitoring room observations for detection. This solution is not only labor-intensive and time-consuming, but also greatly increases labor costs. Furthermore, it cannot achieve 24-hour uninterrupted and comprehensive monitoring without blind spots. This solution is outdated and does not meet the requirements of Industry 4.0 intelligence.

[0004] With the development of technology, image processing techniques are used to collect information from cameras and perform target detection on pedestrian traffic within a given area to detect loitering. While this solution addresses some of the problems associated with traditional manual patrols, it relies solely on simple target detection and cannot acquire the trajectory information of each individual target. Furthermore, this solution fails to meet the practical application requirements for detecting strangers loitering within factories in terms of efficiency and effectiveness.

[0005] Therefore, how to provide a detection method that can achieve 24-hour uninterrupted, comprehensive identification without blind spots, while ensuring the accuracy of identification, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method, apparatus, device and storage medium for detecting the presence of strangers to overcome or at least partially solve the above problems.

[0007] This invention provides the following solution:

[0008] A method for detecting stranger loitering includes:

[0009] The trained object detection network is used to detect pedestrians in multiple consecutive frames of target images, and several pedestrian detection boxes are obtained; the target images are images of the factory target area obtained by an image capturing device;

[0010] The BYTE_Track method is used to divide several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels. The Kalman filter algorithm is used to match the high-confidence detection boxes and the low-confidence detection boxes respectively. The continuous motion trajectory information of the pedestrian is obtained based on the matching results.

[0011] Based on prior knowledge and the continuous motion trajectory information, a prediction is made, and the presence of regional lingering behavior is determined based on the prediction result.

[0012] After confirming the presence of loitering behavior, the pedestrian is identified using a facial recognition network, and the identity of the pedestrian is determined based on the identification results.

[0013] Once the person is identified as a stranger, the result is returned to the monitoring system.

[0014] Preferably, the target detection network includes an improved YOLOv8 target detection network, which includes a C2f module, a fusion module, BCELoss classification loss, and CIOULoss location information. The fusion module includes a feature fusion network FPN and a PAN.

[0015] Preferably, the step of using the BYTE_Track method to divide the pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels includes:

[0016] BYTE uses the threshold T high and T low Several pedestrian detection boxes are respectively included in two sets, where the detection score is higher than T. high The result is included in the high-scoring detection box D. h In, higher than T low Below T high The detection results are included in the low-scoring detection box D. l In this process, the high-confidence detection box and the low-confidence detection box are formed.

[0017] Preferably, the step of using the Kalman filter algorithm to perform detection box matching on the high-confidence detection box and the low-confidence detection box respectively includes:

[0018] The pedestrian trajectory of a frame in each time frame is predicted using a Kalman filter, and the new position of each trajectory is predicted to obtain the prediction box T.

[0019] High-resolution detection box Dh The high-resolution detection box D is calculated by first associating it with the predicted box T. h The similarity is calculated based on the IOU value between the bounding box T and the predicted bounding box T. The Hungarian algorithm is used for matching, and small IOUs are filtered out using match_th = 0.8. Unmatched bounding boxes are saved to D. tbd In the middle, the unmatched predicted bounding boxes are saved to T. tbd ;

[0020] The low-scoring detection box D l and T tbd The detection boxes in the dataset undergo a second matching process using the Hungarian algorithm. Small IOUs are filtered out using match_th = 0.5, and low-scoring detection boxes that fail to match are removed. l Delete the unmatched predicted boxes and save them to T. thd-thd middle.

[0021] Preferably: unmatched trajectory T thd-thd Reserve 60 frames; if more than 60 frames are still unmatched, discard them.

[0022] Unmatched D tbd If four consecutive frames exist, the detection box is initialized as a new trajectory, and the remaining D... tbd The detection frame was discarded.

[0023] Preferably: the step of making a prediction based on prior knowledge and the continuous motion trajectory information, and determining whether there is regional lingering behavior based on the prediction result, includes:

[0024] Determine the wandering area based on prior knowledge;

[0025] Calculate the time t_in when a pedestrian enters the loitering area and the time t_out when a pedestrian leaves the loitering area;

[0026] Calculate the difference between t_out and t_in, compare the difference with a dwell time threshold, and determine that the pedestrian is lingering in the area when the difference exceeds the threshold time.

[0027] Preferably: the step of using a facial recognition network to perform facial recognition on the pedestrian, and determining whether the pedestrian is a stranger based on the recognition result, includes:

[0028] The pedestrian detection box is input into the ResNet152 feature network;

[0029] After aligning the channels of the last three effective feature map layers of the ResNet152 feature network with 1*1 convolution using the FPN pyramid structure, feature fusion operation is performed.

[0030] The SSH module is used to extract features, and the receptive field of the feature map is enhanced by introducing contextual information;

[0031] The prior bounding boxes are binary-classified using softmax to determine whether they are human faces. The prior bounding boxes are then adjusted to obtain the predicted human face bounding boxes.

[0032] Regression is performed on the facial landmarks in the prediction box, and the results are filtered by non-maximum suppression (NMS) to obtain the face detection results.

[0033] The key points of the face in the face detection result are corrected by using FAN key point detection technology to obtain the face alignment result;

[0034] The face alignment result is compared with the face database. When the similarity between the face to be identified and all faces in the face database is lower than the face detection threshold, it is determined to be a stranger.

[0035] A stranger loitering detection device, the device comprising:

[0036] The pedestrian detection box acquisition unit is used to detect pedestrians in multiple consecutive frames of target images using a trained target detection network, and obtain several pedestrian detection boxes; the target images are images of the factory target area acquired by an image capturing device;

[0037] The continuous motion trajectory information acquisition unit is used to divide several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to confidence level using the BYTE_Track method, and to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes respectively using the Kalman filter algorithm, and to obtain the continuous motion trajectory information of the pedestrian based on the matching result;

[0038] The loitering behavior determination unit is used to make a prediction based on prior knowledge and the continuous motion trajectory information, and determine whether there is loitering behavior in the area based on the prediction result.

[0039] The stranger identification unit is used to identify the pedestrian's face using a face recognition network after determining that there is loitering behavior, and to determine whether the pedestrian is a stranger based on the identification result;

[0040] The result return unit is used to return the result to the monitoring system after determining that the person is a stranger.

[0041] A stranger loitering detection device, the device including a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the aforementioned stranger loitering detection method according to the instructions in the program code.

[0044] A computer-readable storage medium for storing program code for performing the above-described stranger loitering detection method.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This application provides a method, apparatus, device, and storage medium for detecting stranger loitering. The method offers high accuracy and efficiency. Furthermore, its non-contact solution enables 24 / 7 real-time, full-coverage detection. Deploying via a server or industrial computer, it is adaptable to both factories with existing cameras and those without, offering rapid and convenient deployment. Therefore, this method is highly suitable for smart factory construction under Industry 4.0.

[0047] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0049] Figure 1 This is a flowchart of a stranger loitering detection method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the YOLOV8 detection framework network provided in an embodiment of the present invention;

[0051] Figure 3 This is a detection effect diagram of the detection frame provided in an embodiment of the present invention;

[0052] Figure 4 This is a first schematic diagram of the trajectory update result provided in an embodiment of the present invention;

[0053] Figure 5 This is a second schematic diagram of the trajectory update result provided in an embodiment of the present invention;

[0054] Figure 6This is a third schematic diagram of the trajectory update results provided in an embodiment of the present invention;

[0055] Figure 7 This is the fourth schematic diagram of the trajectory update results provided in this embodiment of the invention;

[0056] Figure 8 This is the fifth schematic diagram of the trajectory update results provided in this embodiment of the invention;

[0057] Figure 9 This is the sixth schematic diagram of the trajectory update results provided in this embodiment of the invention;

[0058] Figure 10 This is a schematic diagram of personnel entering the loitering area provided in an embodiment of the present invention;

[0059] Figure 11 This is a first schematic diagram of personnel loitering in a loitering area provided in an embodiment of the present invention;

[0060] Figure 12 This is a second schematic diagram of personnel loitering in a wandering area provided by an embodiment of the present invention;

[0061] Figure 13 This is a schematic diagram of personnel leaving the loitering area provided in an embodiment of the present invention;

[0062] Figure 14 This is a flowchart of face detection provided in an embodiment of the present invention;

[0063] Figure 15 This is a face detection result image provided in an embodiment of the present invention;

[0064] Figure 16 This is a schematic diagram of the stranger area loitering detection scheme provided in an embodiment of the present invention;

[0065] Figure 17 This is a schematic diagram of a stranger loitering detection device provided in an embodiment of the present invention;

[0066] Figure 18 This is a schematic diagram of a stranger loitering detection device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0068] See Figure 1 This invention provides a method for detecting the presence of strangers, such as... Figure 1 As shown, the method may include:

[0069] S101: The trained target detection network is used to detect pedestrians in multiple consecutive frame target images to obtain several pedestrian detection boxes; the target images are images of the factory target area obtained by an image capturing device;

[0070] Furthermore, the target detection network includes an improved YOLOv8 target detection network, which includes a C2f module, a fusion module, BCELoss classification loss, and CIOULoss location information. The fusion module includes a feature fusion network FPN and a PAN.

[0071] S102: Using the BYTE_Track method, several pedestrian detection boxes are divided into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels. The Kalman filter algorithm is used to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes respectively. The continuous motion trajectory information of the pedestrian is obtained based on the matching results.

[0072] Furthermore, the method of using BYTE_Track to divide the pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels includes:

[0073] BYTE uses the threshold T high and T low Several pedestrian detection boxes are respectively included in two sets, where the detection score is higher than T. high The result is included in the high-scoring detection box D. h In, higher than T low Below T high The test results are included in D l In this process, the high-confidence detection box and the low-confidence detection box are formed.

[0074] The step of using the Kalman filter algorithm to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes includes:

[0075] The pedestrian trajectory of a frame in each time frame is predicted using a Kalman filter, and the new position of each trajectory is predicted to obtain the prediction box T.

[0076] The high-resolution detection box D h The high-resolution detection box D is calculated by first associating it with the predicted box T. h The similarity is calculated based on the IOU value between the bounding box T and the predicted bounding box T. The Hungarian algorithm is used for matching, and small IOUs are filtered out using match_th = 0.8. Unmatched bounding boxes are saved to D. tbdIn the middle, the unmatched predicted bounding boxes are saved to T. tbd ;

[0077] The low-scoring detection box D l and T tbd The detection boxes in the dataset undergo a second matching process using the Hungarian algorithm. Small IOUs are filtered out using match_th = 0.5, and low-scoring detection boxes that fail to match are removed. l Delete them as background and save the unmatched predicted boxes to T. thd-thd middle.

[0078] Unmatched trajectory T thd-thd Reserve 60 frames; if more than 60 frames are still unmatched, discard them.

[0079] Unmatched D tbd If four consecutive frames exist, the detection box is initialized as a new trajectory, and the remaining D... tbd The detection frame was discarded.

[0080] S103: Make a prediction based on prior knowledge and the continuous motion trajectory information, and determine whether there is regional lingering behavior based on the prediction result;

[0081] Furthermore, the step of making a prediction based on prior knowledge and the continuous motion trajectory information, and determining whether there is regional lingering behavior based on the prediction result, includes:

[0082] Determine the wandering area based on prior knowledge;

[0083] Calculate the time t_in when a pedestrian enters the loitering area and the time t_out when a pedestrian leaves the loitering area;

[0084] Calculate the difference between t_out and t_in, compare the difference with a dwell time threshold, and determine that the pedestrian is lingering in the area when the difference exceeds the threshold time.

[0085] S104: After confirming the presence of loitering behavior, a facial recognition network is used to perform facial recognition on the pedestrian, and the pedestrian is determined to be a stranger based on the recognition result; further, the step of using a facial recognition network to perform facial recognition on the pedestrian and determining whether the pedestrian is a stranger based on the recognition result includes:

[0086] The pedestrian detection box is input into the ResNet152 feature network;

[0087] After aligning the channels of the last three effective feature map layers of the ResNet152 feature network with 1*1 convolution using the FPN pyramid structure, feature fusion operation is then performed.

[0088] The SSH module is used to extract features, and the receptive field of the feature map is enhanced by introducing contextual information;

[0089] The prior bounding boxes are binary-classified using softmax to determine whether they are human faces. The prior bounding boxes are then adjusted to obtain the predicted human face bounding boxes.

[0090] Regression is performed on the facial landmarks in the prediction box, and the results are filtered by non-maximum suppression (NMS) to obtain the face detection results.

[0091] The key points of the face in the face detection result are corrected by using FAN key point detection technology to obtain the face alignment result;

[0092] The face alignment result is compared with the face database. When the similarity between the face to be identified and all faces in the face database is lower than the face detection threshold, it is determined to be a stranger.

[0093] S105: After confirming that the person is a stranger, return the result to the monitoring system.

[0094] The stranger loitering detection method provided in this application embodiment can be used in a factory environment. It combines target detection, multi-target tracking, and face recognition technologies from deep learning image processing to detect stranger loitering within the factory area, outputting the activity trajectory of loitering personnel and triggering an alarm. The method involves processing video stream information collected by cameras within the factory area, converting the video stream into RGB images, and then using the YOLOv8 target detection algorithm to detect personnel targets in the cameras. Next, the BYTE_Track (a target detection-based tracking algorithm) network from multi-target detection is used to match personnel to the detected bounding boxes and obtain the target's tracking trajectory. Prior knowledge is then used to determine if the target is loitering. Finally, a combination of RetinaFace (a face detection network) face detection and ArcFace face recognition is used to determine whether the target's face is that of a stranger. The method provided in this application embodiment is particularly suitable for use in confidential factory environments.

[0095] The methods provided in the embodiments of this application will be described in detail below.

[0096] The method provided in this application deploys cameras within the factory area, collects camera data, decodes it into video streams, and uses deep learning image algorithms to manage and monitor personnel. This overcomes the problem of manual methods not being able to achieve real-time full coverage. Installing cameras within the factory area reduces labor costs, although this is limited by the size of the factory. The method employs YOLOv8 for personnel target detection, then uses the BYTE_Track network in multi-target detection to match personnel to the detected bounding boxes and obtain the target's tracking trajectory. Prior knowledge is then used to determine if the target is loitering, overcoming the limitations of simple target detection schemes in achieving fine-grained multi-target judgment and low recognition efficiency and accuracy. Finally, a combination of RetinaFace face detection and ArcFace face recognition is used to determine if the target's face is that of a stranger, completing the entire process. The method provided in this application achieves real-time, comprehensive, and blind-spot-free recognition while maintaining high accuracy, making it an excellent solution for factory personnel monitoring under Industry 4.0.

[0097] This application provides a solution for detecting lingering strangers in factory areas by utilizing deep learning methods such as object detection, multi-object tracking, and face recognition, combined with prior knowledge. To implement this solution, the following steps are required: Figure 16 As shown, the specific steps are as follows:

[0098] Step 1: In the factory environment, acquire image and video stream information from various locations using cameras, establish a target detection network, perform target detection on pedestrians in the images, and obtain the target detection boxes for pedestrians.

[0099] An improved YOLOv8 object detection system for factory areas was constructed. First, pedestrian samples were collected from factory cameras. Then, annotation tools were used to label the detected pedestrians. Next, an object detection framework based on the YOLOv8 framework was established. In the improved system, the Backbone module replaced the C3 module with the C2f module to achieve a lightweight basic feature extraction scheme. The fusion module used a feature fusion network (FPN) and a PAN to generate pedestrian detection boxes (Anchors). Overlapping redundant Anchors were filtered using NMS. BCELoss was used as the classification loss, and CIOULoss was used as the location information loss. Finally, the pedestrian detection boxes (Bboxes) were obtained. The network was trained using the monitoring dataset to generate the prediction model. Video image information parsed from the factory cameras was used as input to the model to acquire pedestrian object detection boxes in the factory area.

[0100] In step one, firstly, within the factory environment, image video stream information from various locations is acquired using cameras. A target detection network is then established to perform target detection on pedestrians in the images, obtaining the target detection bounding boxes for each pedestrian. The specific implementation steps are as follows:

[0101] Step 101: First, establish a target detection dataset for factory pedestrians. Using factory surveillance videos and relevant sample images of pedestrians from the Internet, label the pedestrian parts in these images using a labeling tool. The labeling format adopts an improved VOC format. After labeling, combine the above images with the COCO and VOC datasets to form an enhanced factory pedestrian detection VOC dataset.

[0102] Step 102: Based on step 101, establish the YOLOv8 structure. The framework structure diagram is as follows: Figure 2 As shown, the YOLOv8 framework backbone first adopts the CSP construction idea, achieving lightweight processing by replacing the C3 module with the C2f module. The C2f module utilizes the ElAN concept for design improvements, changing the number of blocks from 3-6-9-3 to 3-6-6-3, employing more skip connections and more split operations to achieve lightweight processing. Simultaneously, the kenel of the first convolutional layer is changed from 6*6 to 3*3. This operation achieves the optimal feature extraction module. The fusion module uses the feature fusion network FPN and PAN to generate a combined anchor structure of keypoints and detection boxes. Then, NMS is used to filter redundant anchors, BCELoss is used as the classification loss, and CIOULoss is used as the location information loss. The detection box CIOU_loss is shown in Equation 1, and the classification CLS_loss is shown in Equation 2. Multi-dimensional regression is used to determine the category and location of the detection boxes. Then, a joint loss update mechanism is used to adjust the various training parameters, finally completing the training of the entire network.

[0103] CIOU_Loss=1-IOU(t′,t) (Formula 1)

[0104] Where t′ represents GTbbox information.

[0105] CLs_Loss=BCE(c′ n ,c n (Equation 2)

[0106] BCE=―((c n *logc′ n )+(1―c n )*log(1―c′ n (Equation 3)

[0107] Step 103: The object detection dataset from Step 101 is used as the training set for the YOLOV8 object detection framework network in Step 102. The network is trained on two 3080ti graphics cards for a total of 400 epochs. The final trained YOLOV8 large model achieves a bounding box regression accuracy of 99.6% and a recall of 99.7%.

[0108] Step 104: Using the network from step 102 in conjunction with the model obtained in step 103, target detection and keypoint prediction are performed. First, the factory camera images need to be encoded and decoded. FFMPEG is used to decode the camera's video stream information into RGB images as a prediction buffer. Then, the buffer information is inspected for frame stillness, black levels, and sharpness. After confirming there are no such issues, forward prediction is performed using the trained model file. Based on the prediction results, pedestrian detection boxes for employees are obtained, preparing for subsequent multi-object tracking and stranger detection. The detection results are as follows: Figure 3 As shown.

[0109] Step 2: For the detection boxes detected in Step 1, use the continuous frame information of the video stream to classify all pedestrian target detection boxes, use BYTE to associate high-scoring detection boxes and low-scoring detection boxes, and then use the Kalman filter algorithm to match them to obtain the continuous motion trajectory of the corresponding pedestrian.

[0110] In specific implementation, a BYTE_Track trajectory tracking network is constructed. First, the BYTE method is used to divide pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels. A two-stage matching method is used to match the detection boxes. The first stage uses a Kalman filter to perform trajectory matching on the high-confidence detection boxes, and the second stage uses the same method to perform trajectory matching on the low-confidence detection boxes. The two-stage matching scheme can solve problems such as occlusion and omission in the video. After matching, the trajectory routes of all people under the camera can be obtained.

[0111] First, the detection boxes obtained in Step 1 are marked. Then, using continuous frame information from the video stream, all pedestrian target detection boxes are graded. High-scoring and low-scoring detection boxes are associated using a BYTE algorithm. Finally, a Kalman filter algorithm is used for matching to obtain the continuous motion trajectory of the corresponding pedestrian. The specific implementation steps of Step 2 are as follows:

[0112] Step 201: The input video sequence is denoted as V, and the detection result for each frame in V is denoted as D. Since each detection result has a detection score, BYTE sets a threshold T. high and T low The detection results are counted into two sets, with the set containing the detection score higher than T. high The result is included in the high-scoring detection box D.h In, higher than T low Below T high The test results are included in D l In the process, high-confidence detection boxes and low-confidence detection boxes are formed.

[0113] After separating the detection boxes in steps 202 and 201, the pedestrian trajectory of one frame in each time frame is predicted using a Kalman filter to predict the new position of each trajectory and obtain the prediction box T.

[0114] Step 203, the high-resolution detection box D obtained in step 201 is... h The predicted bounding box T obtained in step 202 is used for the first association process, and the high-resolution detection bounding box D is calculated. h The similarity is calculated from the IOU value between the predicted bounding box T and the target bounding box T. Then, the Hungarian algorithm is used for matching, and the smaller IOUs are filtered out using match_th = 0.8. The detection results that are not matched are saved to D. tbd In the middle, the unmatched predicted bounding boxes are saved to T. tbd .

[0115] Step 204, the low-resolution detection box D obtained in step 201 is... l And T obtained in step 203 tbd A second matching process is performed. Since low-scoring detection boxes may have lower scores due to motion blur or occlusion, the IOU similarity threshold in the second matching needs to be lower than that in the first matching. The Hungarian algorithm is used for matching again, and a match_th = 0.5 is used to filter out low-scoring IOUs. After completion, the low-scoring detection boxes that still fail to match are D. l Delete them as background and save the unmatched predicted boxes to T. thd-thd This is in preparation for subsequent processing.

[0116] Step 205, for the trajectory T that was not matched in step 204 thd-thd The missing detection boxes may be due to occlusion or motion blur, so these unmatched tracks are retained for 60 frames. If they are still not matched after 60 frames, they are discarded.

[0117] Step 206, remove the unmatched D from step 203. tbd If four consecutive frames exist, the detection box is initialized as a new trajectory, and the remaining D... tbd The detection box is discarded, thus completing the entire trajectory tracking process for video sequence V. The trajectory update results are as follows: Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 As shown.

[0118] Step 3: Based on prior knowledge and the pedestrian trajectory information obtained in Step 2, make a prediction and determine whether there is any lingering behavior in the area based on the results.

[0119] In practice, a priori knowledge-based lingering personnel detection system is constructed. Based on the personnel trajectory obtained in the second step, the lingering area is defined, and the lingering time of the trajectory in the lingering area is used to determine whether the personnel in the trajectory are lingering in the area.

[0120] Based on prior knowledge and the pedestrian trajectory information obtained in step two, a prediction is made, and the existence of lingering behavior in the area is determined based on the results. The specific implementation steps of step three are as follows:

[0121] Step 301: Based on the motion trajectories of each target in V obtained in Step 2, the loitering area is first determined using prior knowledge. This area is pre-deployed in the algorithm by the system. Then, the time t_in when the target enters the loitering area is calculated, such as... Figure 10 As shown, people then lingered and loitered within the area, as... Figure 11 , Figure 12 As shown, the departure time t_out is recorded after the target leaves the area. Figure 13 As shown, the difference between t_out and t_in is calculated and compared with the stay time threshold. If the stay time exceeds the threshold, it is determined that the person has stayed in this area.

[0122] Step 4: Perform facial recognition on pedestrians lingering in the area identified in Step 3, establish a facial recognition network, and if a stranger is detected lingering in the area, an alarm will be triggered and the information will be uploaded to the monitoring system.

[0123] In practice, a stranger recognition network is constructed. The third step, after detecting loitering, is to perform face recognition on the loitering person. The RentinaFace face detection scheme is used to detect the loitering person's face. Then, the FAN keypoint detection technology is used to detect the key points of the face in the face detection box. Next, the keypoint information is used for face correction. Finally, the metric learning method with ArcFaceLoss as the loss function is used to complete the face recognition. Based on the result, it is determined whether the person is a stranger.

[0124] Step four involves performing facial recognition on pedestrians lingering in the areas identified in step three, establishing a facial recognition network. If a stranger is detected lingering in an area, an alarm is triggered, and the information is uploaded to the monitoring system. The specific implementation steps of step four are as follows:

[0125] Step 401: Based on the results of step 301, when any ID trajectory shows lingering behavior, face detection is performed on the corresponding detection box results from step 104 using the RetinaFace face detection framework. The specific process is as follows: Figure 14 As shown, the detection bounding box from step 104 is first used as input. ResNet152 is used to extract features from the input image. Then, using the FPN pyramid structure, the last three effective feature map layers of ResNet152 are aligned using 1*1 convolutions before feature fusion. Next, the SSH module is used to further extract features, primarily by introducing contextual information to enhance the receptive field of the feature map. Finally, softmax is used to perform binary classification on the prior bounding box anchors to determine if it is a face. The prior bounding boxes are then adjusted to obtain the predicted face bounding box. Finally, facial landmarks within the predicted bounding box are regressed. After filtering the results using NMS (Non-Maximum Suppression), the final face detection result is obtained, as shown below. Figure 15 As shown.

[0126] Step 402, Step 401: The face detection results use FAN key point detection technology to correct and align the key points of the face in the face detection box, in preparation for subsequent face recognition.

[0127] In step 403, after obtaining the face alignment result in step 402, Arcface is used to train the face recognition metric to learn the feature comparison scheme and establish a face database for the factory area. The face alignment result in step 402 is compared with the face database. Since the face is greatly affected by the angle, multiple faces in the lingering trajectory are identified. When the similarity between the face to be identified and the face in the face database is lower than the face detection threshold, it is judged to be a stranger.

[0128] Step 404: When a stranger is detected in the loitering trajectory in step 403, the alarm procedure is activated, a signal is transmitted to the alarm device, and the result is transmitted back to the monitoring system to remind the management personnel to handle the situation in a timely manner.

[0129] The above four steps summarize the entire solution. This method collects video from factory cameras, processes the video using deep learning algorithms, and combines this with prior knowledge verification to ultimately implement the function of detecting unfamiliar personnel loitering in the factory area. This method overcomes the problems of low accuracy, low efficiency, and high labor costs associated with traditional solutions. Furthermore, this method has the ability to continuously learn and optimize, exhibiting strong generalization capabilities. The personnel trajectory information and facial information provided by this method, through different combinations, can facilitate personnel management for factory managers.

[0130] In summary, the stranger lingering detection method provided in this application features high detection accuracy and excellent detection efficiency. Furthermore, this method employs a non-contact solution, enabling contactless, full-coverage, 24 / 7 real-time detection. It also utilizes a standalone server or industrial control computer deployment method, making it adaptable to both factories with existing cameras and those without. Therefore, this method is characterized by rapid and convenient deployment. Clearly, this method is highly adaptable to the construction of smart factories under Industry 4.0.

[0131] See Figure 17 This application embodiment can also provide a stranger loitering detection device, such as... Figure 17 As shown, the device may include:

[0132] The pedestrian detection box acquisition unit 1701 is used to detect pedestrians in multiple consecutive frames of target images using a trained target detection network to obtain several pedestrian detection boxes; the target images are images of the factory target area acquired by an image capturing device.

[0133] The continuous motion trajectory information acquisition unit 1702 is used to divide a number of pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to confidence level using the BYTE_Track method, and to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes respectively using the Kalman filter algorithm, and to obtain the continuous motion trajectory information of the pedestrian based on the matching result.

[0134] The loitering behavior determination unit 1703 is used to make a prediction based on prior knowledge and the continuous motion trajectory information, and determine whether there is loitering behavior in the area based on the prediction result.

[0135] The stranger identification unit 1704 is used to identify the pedestrian's face using a face recognition network after determining that there is loitering behavior, and to determine whether the pedestrian is a stranger based on the identification result.

[0136] Result return unit 1705 is used to return the result to the monitoring system after determining that the person is a stranger.

[0137] This application embodiment can also provide a stranger loitering detection device, the device including a processor and a memory:

[0138] The memory is used to store program code and transmit the program code to the processor;

[0139] The processor is used to execute the steps of the stranger loitering detection method described above according to the instructions in the program code.

[0140] like Figure 18As shown in the figure, the stranger loitering detection device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0141] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0142] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the stranger loitering detection method.

[0143] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0144] The trained object detection network is used to detect pedestrians in multiple consecutive frames of target images, and several pedestrian detection boxes are obtained; the target images are images of the factory target area obtained by an image capturing device;

[0145] The BYTE_Track method is used to divide several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels. The Kalman filter algorithm is used to match the high-confidence detection boxes and the low-confidence detection boxes respectively. The continuous motion trajectory information of the pedestrian is obtained based on the matching results.

[0146] Based on prior knowledge and the continuous motion trajectory information, a prediction is made, and the presence of regional lingering behavior is determined based on the prediction result.

[0147] After confirming the presence of loitering behavior, the pedestrian is identified using a facial recognition network, and the identity of the pedestrian is determined based on the identification results.

[0148] Once the person is identified as a stranger, the result is returned to the monitoring system.

[0149] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0150] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0151] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0152] Of course, it should be noted that, Figure 18 The structure shown does not constitute a limitation on the stranger loitering detection device in the embodiments of this application. In practical applications, the stranger loitering detection device may include more than Figure 18 More or fewer components as shown, or combinations of certain components.

[0153] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described stranger loitering detection method.

[0154] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0157] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting the presence of strangers, characterized in that, include: The trained object detection network is used to detect pedestrians in multiple consecutive frames of object images, and several pedestrian detection boxes are obtained. The target image is an image of the target area of ​​the factory obtained through an image capture device; The BYTE_Track method is used to divide several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to their confidence levels. The Kalman filter algorithm is used to match the high-confidence detection boxes and the low-confidence detection boxes respectively. The continuous motion trajectory information of the pedestrian is obtained based on the matching results. The BYTE_Track method is used to classify several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes based on their confidence levels, including: BYTE uses the threshold T high and T low Several pedestrian detection boxes are respectively included in two sets, where the detection score is higher than T. high The result is included in the high-scoring detection box D. h In, higher than T low Below T high The detection results are included in the low-scoring detection box D. l In the process, the high-confidence detection box and the low-confidence detection box are formed; The step of using the Kalman filter algorithm to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes includes: The pedestrian trajectory of a frame in each time frame is predicted using a Kalman filter, and the new position of each trajectory is predicted to obtain the prediction box T. The high-resolution detection box D h The high-resolution detection box D is calculated by first associating it with the predicted box T. h The similarity is calculated from the IOU value between the predicted bounding box T and the predicted bounding box T. The Hungarian algorithm is used for matching, and small IOUs are filtered out using match_th=0.

8. Unmatched bounding boxes are saved to D. tbd In the middle, the unmatched predicted bounding boxes are saved to T. tbd ; The low-scoring detection box D l and T tbd The detection boxes in the dataset undergo a second matching process using the Hungarian algorithm. Small IOUs are filtered out using match_th=0.5, and low-scoring detection boxes that fail to match are removed. l Delete the unmatched predicted boxes and save them to T. thd-thd middle; Based on prior knowledge and the continuous motion trajectory information, a prediction is made, and the presence of regional lingering behavior is determined based on the prediction result. After confirming the presence of loitering behavior, the pedestrian is identified using a facial recognition network, and the identity of the pedestrian is determined based on the identification results. Once the person is identified as a stranger, the result is returned to the monitoring system.

2. The stranger loitering detection method according to claim 1, characterized in that, The target detection network includes an improved YOLOv8 target detection network, which includes a C2f module, a fusion module, BCELoss classification loss, and CIOULoss location information. The fusion module includes a feature fusion network FPN and a PAN.

3. The stranger loitering detection method according to claim 1, characterized in that, Unmatched trajectory T thd-thd Reserve 60 frames; if more than 60 frames are still unmatched, discard them. Unmatched D tbd If four consecutive frames exist, the detection box is initialized as a new trajectory, and the remaining D... tbd The detection frame was discarded.

4. The stranger loitering detection method according to claim 1, characterized in that, The step of making a prediction based on prior knowledge and the continuous motion trajectory information, and determining whether there is regional lingering behavior based on the prediction result, includes: Determine the wandering area based on prior knowledge; Calculate the time t_in when a pedestrian enters the loitering area and the time t_out when a pedestrian leaves the loitering area; Calculate the difference between t_out and t_in, compare the difference with a dwell time threshold, and determine that the pedestrian is lingering in the area when the difference exceeds the threshold time.

5. The stranger loitering detection method according to claim 1, characterized in that, The step of using a facial recognition network to perform facial recognition on the pedestrian, and determining whether the pedestrian is a stranger based on the recognition result, includes: The pedestrian detection box is input into the ResNet152 feature network; After aligning the channels of the last three effective feature map layers of the ResNet152 feature network with 1*1 convolution using the FPN pyramid structure, feature fusion operation is performed. The SSH module is used to extract features, and the receptive field of the feature map is enhanced by introducing contextual information; The prior bounding boxes are binary-classified using softmax to determine whether they are human faces. The prior bounding boxes are then adjusted to obtain the predicted human face bounding boxes. Regression is performed on the facial landmarks in the prediction box, and the results are filtered by non-maximum suppression (NMS) to obtain the face detection results. The key points of the face in the face detection result are corrected by using FAN key point detection technology to obtain the face alignment result; The face alignment result is compared with the face database. When the similarity between the face to be identified and all faces in the face database is lower than the face detection threshold, it is determined to be a stranger.

6. A stranger loitering detection device, characterized in that, The apparatus for performing the stranger loitering detection method according to any one of claims 1-5, the apparatus comprising: The pedestrian detection box acquisition unit is used to detect pedestrians in multiple consecutive frames of target images using a trained target detection network, and obtain several pedestrian detection boxes; the target images are images of the factory target area acquired by an image capturing device; The continuous motion trajectory information acquisition unit is used to divide several pedestrian detection boxes into high-confidence detection boxes and low-confidence detection boxes according to confidence level using the BYTE_Track method, and to perform detection box matching on the high-confidence detection boxes and the low-confidence detection boxes respectively using the Kalman filter algorithm, and to obtain the continuous motion trajectory information of the pedestrian based on the matching result; The loitering behavior determination unit is used to make a prediction based on prior knowledge and the continuous motion trajectory information, and determine whether there is loitering behavior in the area based on the prediction result. The stranger identification unit is used to identify the pedestrian's face using a face recognition network after determining that there is loitering behavior, and to determine whether the pedestrian is a stranger based on the identification result; The result return unit is used to return the result to the monitoring system after determining that the person is a stranger.

7. A stranger loitering detection device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the stranger loitering detection method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the stranger loitering detection method according to any one of claims 1-5.

Images