Personnel off-post detection method and device, storage medium and computer program product

Abstract

The invention discloses a personnel off-post detection method and device, a storage medium and a computer program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a current frame monitoring image in a target monitoring scene, and transmitting the current frame monitoring image to a YOLO target detection model to obtain a first image personnel number; if the number of the first image personnel is smaller than the preset number of post personnel, detecting whether the post lacking duration reaches the preset duration or not; if the off-duty duration reaches the preset duration, detecting the current frame of monitoring image through a multi-modal target detection model to obtain a second image personnel number; and if the number of personnel in the second image is smaller than the number of post personnel, determining that personnel leaving exists in the target monitoring scene. According to the method and the device, the high efficiency of the YOLO target detection model and the high accuracy of the multi-modal target detection model are combined, so that the relatively high detection efficiency is maintained, and the accuracy of personnel off-post detection is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to methods, devices, storage media and computer program products for detecting personnel leaving their posts. Background Art

[0002] Off-duty behavior detection is a type of motion recognition, which is used to identify and detect off-duty behavior of personnel in specific scenarios. It captures video streams of key areas by installing cameras and uses AI image analysis technology to process and analyze video content in real time. It is usually used in places where close attention needs to be paid to the working status and behavior of personnel, such as factories, construction sites, mines, offices, monitoring rooms, etc.

[0003] In the existing off-duty detection technology, the human body in the image is first detected by the target detection algorithm, and then the human body is analyzed. At present, the YOLO (You Only Look Once) series and R-CNN (Region-based Convolutional Neural Networks) series detection models are usually used to detect the human body. However, although these small target detection networks have a fast detection speed, the model is limited by the training results of the data set, and it is easy to miss the detection when detecting the human body, which affects the accuracy of off-duty detection. In recent years, the accuracy of human body detection can be improved by introducing large multimodal models. However, the processing time of large multimodal models is long, which affects the detection efficiency of off-duty detection. In summary, there is a problem of unbalanced accuracy and detection efficiency in the current off-duty detection process.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a method, device, storage medium and computer program product for detecting personnel leaving their posts, aiming to solve the technical problem of the inability to balance accuracy and detection efficiency in the current personnel leaving their posts detection process.

[0006] To achieve the above purpose, the present application proposes a method for detecting personnel leaving their posts, the method comprising:

[0007] Acquire a current frame monitoring image in a target monitoring scene, and send the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image;

[0008] Based on the number of personnel in the first image, detecting whether the absence duration has reached a preset duration, wherein the absence duration is a continuous duration during which the number of personnel in the first image is less than the number of personnel in the post;

[0009] If the absence duration reaches the preset duration, the current frame monitoring image is input into a preset multimodal target detection model to obtain the number of personnel in the second image of the current frame monitoring image;

[0010] If the number of personnel in the second image is less than the number of personnel at the post, it is determined that there are personnel leaving their posts in the target monitoring scene.

[0011] In one embodiment, the step of detecting whether the absence duration has reached a preset duration based on the number of personnel in the first image includes:

[0012] If the number of people in the first image is less than the preset number of people at the post, the target number of frames is cumulatively added by one, and when the target number of frames is greater than or equal to the preset number of frames, it is determined that the absence duration reaches the preset duration, wherein the target number of frames is the number of consecutive frames of the surveillance image in which the number of people in the image is less than the number of people at the post;

[0013] If the number of people in the first image is greater than or equal to the number of people in the position, the target frame number is cleared, and the process returns to the step of obtaining the current frame monitoring image under the target monitoring scene, and sending the current frame monitoring image to the preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image.

[0014] In one embodiment, after the step of inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame surveillance image, the method further includes:

[0015] If the number of people in the second image is greater than or equal to the number of people in the position, the target number of frames is cleared.

[0016] In one embodiment, the step of sending the current frame surveillance image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame surveillance image includes:

[0017] Inputting the current frame monitoring image into the YOLO target detection model to obtain the detection frame coordinates of multiple detection frames and the confidence corresponding to each of the detection frames;

[0018] Determine, from each of the detection frames, a target frame whose confidence is greater than or equal to a preset confidence threshold;

[0019] Traversing each of the target frames in descending order of confidence, determining the currently traversed target frame as the current frame, and calculating the intersection and union ratio of each non-current frame with the current frame, wherein the non-current frame is a detection frame in each of the target frames except the current frame;

[0020] If the intersection-over-union ratio is less than or equal to a preset intersection-over-union ratio threshold, the object corresponding to the current frame is regarded as the inspected person;

[0021] The number of the inspected persons is determined, and the number of the inspected persons is determined as the number of persons in the first image in the current frame monitoring image.

[0022] In one embodiment, before the step of sending the current frame surveillance image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame surveillance image, it also includes:

[0023] Scaling the current frame monitoring image until the length of the long border of the current frame monitoring image reaches a first preset side length to obtain a first processed image;

[0024] Adjusting the length of the short border of the first processed image to be consistent with the first preset side length to obtain a second processed image;

[0025] Adjusting the image format of the second processed image to a preset YOLO model input format to obtain a third processed image;

[0026] The third processed image is converted into a tensor image to obtain a fourth processed image, and each pixel value in the fourth processed image is normalized to obtain a first target image. The first target image is used as the current frame monitoring image, and the step of sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image is executed.

[0027] In one embodiment, the multimodal object detection model includes a text encoder, an image encoder, and an object detection module;

[0028] The step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame monitoring image includes:

[0029] Obtaining text description information corresponding to the current frame surveillance image, wherein the text description information is used to describe the person in the current frame surveillance image;

[0030] Inputting the current frame monitoring image into the image encoder to obtain an image feature vector, and inputting the text description information into the text encoder to obtain a text feature vector;

[0031] Fusion of the image feature vector and the text feature vector to obtain a depth feature vector;

[0032] The depth feature vector is input into the target detection module to obtain the number of people in the second image in the current frame monitoring image.

[0033] In one embodiment, before the step of inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame surveillance image, the step includes:

[0034] Converting the current frame monitoring image into a tensor image to obtain a fifth processed image;

[0035] Scaling the fifth processed image until the length of the short border of the fifth processed image reaches a second preset side length, thereby obtaining a sixth processed image;

[0036] Adjusting the image format of the sixth processed image to a preset multimodal model input format to obtain a seventh processed image;

[0037] The pixel values ​​of each pixel in the seventh processed image are standardized to obtain a second target image, the second target image is used as the current frame monitoring image, and the step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame monitoring image is executed.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a personnel absence detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the personnel absence detection method as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the personnel leaving the post detection method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the personnel absence detection method described above are implemented.

[0041] In the present application, a current frame surveillance image in a target surveillance scene is obtained, and the current frame surveillance image is sent to a preset YOLO target detection model to obtain the number of people in the first image in the current frame surveillance image; if the number of people in the first image is less than the preset number of people in the post, it is detected whether the absence duration reaches the preset duration, wherein the absence duration is the continuous duration during which the number of people in the first image is less than the number of people in the post; if the absence duration reaches the preset duration, the current frame surveillance image is input into the preset multimodal target detection model to obtain the number of people in the second image in the current frame surveillance image; if the number of people in the second image is less than the number of people in the post, it is determined that there are people leaving their posts in the target surveillance scene.

[0042] In this application, by combining the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, the YOLO target detection model is used to detect human bodies in the current frame monitoring image, and a large number of video frames can be processed in a short time, thereby ensuring the detection efficiency. The current frame monitoring image is further detected by the multimodal target detection model, and the multimodal model uses more information to improve the accuracy of human body detection. When the multimodal target detection model also detects that the number of personnel is insufficient, it is determined that there are personnel leaving their posts, thereby improving the accuracy of personnel leaving their posts detection.

[0043] In addition, during the personnel leaving their posts detection process, continuous frame detection reduces the number of missed detections by the YOLO target detection model, further improving the accuracy of personnel leaving their posts detection. Moreover, continuous frame detection is performed based on the output of the YOLO model, and does not require additional detection operations, which reduces the complexity of detection and thus avoids affecting the personnel leaving their posts detection.

[0044] Therefore, by combining the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, the accuracy of personnel leaving their posts is improved while maintaining a high detection efficiency, solving the problem of the inability to balance accuracy and detection efficiency in the current personnel leaving their posts detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1A flow chart of the first embodiment of the present invention for detecting the personnel leaving their posts;

[0048] Figure 2 A brief flowchart of a method for detecting personnel leaving their post provided in one embodiment of the present application;

[0049] Figure 3 An image preprocessing process corresponding to the YOLO target detection model of the personnel leaving the post detection method provided in an embodiment of the application;

[0050] Figure 4 A schematic diagram of frame intersection and frame union involved in a method for detecting personnel leaving their posts provided in an embodiment of the application;

[0051] Figure 5 An image preprocessing process corresponding to a multimodal target detection model of a method for detecting personnel leaving their post provided in an embodiment of the application;

[0052] Figure 6 A multimodal target detection model processing flow for a personnel absence detection method provided in an embodiment of the application;

[0053] Figure 7 Schematic diagram of the equipment structure of the hardware operating environment involved in the personnel absence detection method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: obtain the current frame surveillance image in the target monitoring scene, and send the current frame surveillance image to the preset YOLO target detection model to obtain the number of people in the first image in the current frame surveillance image; if the number of people in the first image is less than the preset number of people in the post, detect whether the absence duration reaches the preset duration, wherein the absence duration is the continuous duration that the number of people in the first image is less than the number of people in the post; if the absence duration reaches the preset duration, input the current frame surveillance image into the preset multimodal target detection model to obtain the number of people in the second image in the current frame surveillance image; if the number of people in the second image is less than the number of people in the post, it is determined that there are people leaving their posts in the target monitoring scene.

[0058] In this embodiment, for the convenience of description, the following description is made with the personnel leaving the post detection device as the execution subject.

[0059] Action recognition is a very important and basic task in the field of computer vision. It aims to identify and understand the actions of people or objects in videos or image sequences by analyzing them. Off-duty behavior detection is a type of action recognition, which is used to identify and detect off-duty behavior of personnel in specific scenarios. It captures video streams of key areas by installing cameras and uses AI (Artificial Intelligence) image analysis technology to process and analyze video content in real time. It is usually used in places where people's working status and behavior need to be highly concerned, such as factories, construction sites, mines, offices, monitoring rooms, etc. At present, off-duty detection technology detects the human body in the image through the target detection algorithm and then analyzes the human body. Therefore, the effect of human body detection has an important impact on the effect of off-duty behavior detection. At present, the YOLO series and R-CNN series detection models are usually used to detect human bodies. These open-source small object detection networks are pre-trained on the COCO (Common Objects in Context) dataset to achieve human body detection. Although the detection speed is very fast and widely used in actual scenarios, due to the limitation of the dataset, the detection effect of the model is affected by factors such as shooting angle, lighting, and occlusion. When detecting human bodies, missed detection often occurs. When detecting human bodies, missed detection is easy to occur, which affects the accuracy of personnel leaving their posts detection. In recent years, with the rise and development of multimodal and large model technologies, the processing of computer vision tasks has been greatly improved. Among them, the target detection algorithm has greatly improved the detection of various objects by introducing a new modality - text information. In human body detection, although the large model shows better detection effect than the small model, the large model is slow and the processing time for leaving the post detection is long, which affects the detection efficiency of personnel leaving the post detection. In summary, there is a problem that the accuracy and detection efficiency cannot be balanced in the current personnel leaving the post detection process.

[0060] At present, there are some methods for off-post detection. For example, AI intelligent detection of safe production can be performed based on video analysis and computer vision, and a target detection model based on a convolutional neural network can be used to detect human bodies in designated production areas. The focus is on calculating the safety detection degree assessment coefficient based on the results of human body detection, screening data through the calculated safety coefficient, and recording abnormal changes. However, the human body detection process has not been optimized and improved, and the safety coefficient is calculated based on a single frame image, without considering occlusion, lighting and other conditions in the actual scene, which is prone to misjudgment and the accuracy of off-post detection is low. For example, off-post detection can be performed based on a secondary target tracking algorithm, specifically using a target tracking algorithm based on the YOLOv5 algorithm and a target tracking algorithm, by analyzing and classifying the motion trajectory and behavior of the target person, it is determined whether the person is off-post, and the detection accuracy is improved by performing two target tracking operations. However, the target tracking algorithm is relatively complex and time-consuming, and is easily affected by occlusion and changes in the appearance of the target person. It is difficult to apply to scenes with high real-time requirements, and the accuracy of off-post detection is low. For example, a target tracking algorithm based on the Hungarian algorithm and P2PNet (Peer-to-Peer In this paper, a point-to-point neural network (P2PNet) is used to detect personnel leaving their posts. The P2PNet algorithm is used to detect human heads and count the number of people in the current area. When the number of people is abnormal, the Hungarian matching algorithm is used to determine the personnel leaving their posts. However, this method detects single-frame images and is easily interfered by external factors such as occlusion, which affects the detection effect. In other words, the existing personnel leaving their posts detection algorithms can achieve the function of leaving their posts detection to a certain extent through human body detection, human head detection, etc., but there are still some shortcomings, mainly including the following points: (1) Most existing algorithms directly use the results of human body detection for subsequent judgment, and few algorithms are specifically optimized for the human body detection process. However, in practical applications, if the human body detection effect is not good, it is difficult for leaving the post detection to have a good performance. (2) In practical applications, most detection is based on video streams, but most algorithms are based on single-frame images for judgment. This not only fails to give full play to the advantages of video streams, but also single-frame images may have occlusion, blur, etc., which affects the judgment results. (3) The actual application scenarios have high real-time requirements, and more complex algorithms, such as human body tracking and face recognition, are difficult to be widely used in actual scenarios.

[0061] This application provides a solution, which combines the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, and uses the YOLO target detection model to detect human bodies in the current frame monitoring image, so that a large number of video frames can be processed in a short time, thereby ensuring the detection efficiency. The current frame monitoring image is further detected by the multimodal target detection model, and the multimodal model uses more information to improve the accuracy of human body detection. When the multimodal target detection model also detects that the number of personnel is insufficient, it is determined that there are personnel leaving their posts, thereby improving the accuracy of personnel leaving their posts detection.

[0062] In addition, during the personnel leaving their posts detection process, continuous frame detection reduces the number of missed detections by the YOLO target detection model, further improving the accuracy of personnel leaving their posts detection. Moreover, continuous frame detection is performed based on the output of the YOLO model, and does not require additional detection operations, which reduces the complexity of detection and thus avoids affecting the personnel leaving their posts detection.

[0063] Therefore, by combining the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, the accuracy of personnel leaving their posts is improved while maintaining a high detection efficiency, solving the problem of the inability to balance accuracy and detection efficiency in the current personnel leaving their posts detection process.

[0064] Compared with directly using the results of human body detection for subsequent judgment, this application uses the fast and high-precision YOLO target detection model for preliminary human body detection. When the number of people in the first image is less than the preset number of people in the post, that is, when there is a possibility of leaving the post, the absence duration is further detected instead of directly determining it as leaving the post. This can reduce misjudgment caused by human body detection errors, thereby improving the accuracy of leaving the post detection.

[0065] Compared with judgment based on a single-frame image, the advantages of video streams are not brought into play, and single-frame images may have problems such as occlusion and blur. This embodiment realizes real-time processing of video streams by continuously acquiring the current frame monitoring image in the target monitoring scene and using the YOLO model for human body detection. It also uses the temporal continuity of the video stream for judgment, reduces the possible misjudgment caused by a single-frame image, ensures the accuracy of the judgment, and improves the accuracy of the personnel leaving the post detection results.

[0066] Compared with the actual application scenarios with high real-time requirements, more complex algorithms such as human tracking and face recognition are difficult to be widely used. This application uses the fast and real-time YOLO model to perform preliminary human detection to meet the real-time requirements of actual application scenarios. After detecting possible absence from work, a preliminary judgment is made by calculating the absence duration. If the absence duration reaches the preset duration, the solution will use the multimodal target detection model for further verification. Although the multimodal target detection model may be relatively complex, it is only used after the absence duration reaches the preset duration, so it has little impact on the real-time performance of the entire system, thereby improving the real-time performance of personnel absence detection.

[0067] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a personnel leaving the post detection device, etc. The following takes the personnel leaving the post detection device as an example to illustrate this embodiment and the following embodiments.

[0068] Based on this, the present application embodiment provides a method for detecting personnel leaving their posts, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the personnel leaving their posts detection method of the present applicant.

[0069] In this embodiment, the personnel leaving the post detection method includes steps S10 to S40:

[0070] Step S10, obtaining a current frame monitoring image in a target monitoring scene, and sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image;

[0071] It should be noted that the target monitoring scene refers to the specific environment or area where personnel leaving their posts need to be monitored. For example, in the fields of security and production management, specific areas need to be monitored for the purpose of ensuring safety and improving production efficiency. These monitored areas are target monitoring scenes. Target monitoring scenes can include factory workshops, offices, public places and other types, which are not limited here. The current frame monitoring image is a frame of image in the monitoring video collected by the image sensor. In video monitoring, the video stream is composed of multiple continuous frames of images, and the current frame monitoring image refers to a frame of image currently used for personnel leaving their posts detection. The current frame monitoring image is the basis for the monitoring system to perform tasks such as target detection and behavior analysis. By processing and analyzing the current frame image, the recognition, tracking and early warning of the target in the monitoring scene can be realized.

[0072] In this embodiment, the preset YOLO target detection model is used to process the current frame surveillance image to obtain the number of people in the current frame (hereinafter referred to as the number of people in the first image). The YOLO model can simultaneously predict the category and location information of the object in one forward propagation, so it has a faster detection speed and higher detection accuracy.

[0073] The YOLO target detection model divides the input current frame monitoring image into S×S grids, each grid is responsible for predicting the target whose center point falls within the grid, and each grid predicts multiple detection frames and the confidence scores of these detection frames. By combining this information, YOLO can efficiently detect the target in the image. It is understandable that the historical monitoring images in the target monitoring scene can be used as training data, and the number of people in the historical monitoring images can be used as training labels to train the initial YOLO model to obtain the YOLO target detection model. In this embodiment, there is no restriction on the structure and training process of the YOLO target detection model, which can be set according to actual needs.

[0074] Step S20, based on the number of personnel in the first image, detecting whether the absence duration has reached a preset duration, wherein the absence duration is the duration during which the number of personnel in the first image is less than the number of personnel in the post;

[0075] It should be noted that the number of personnel in a position refers to the number of personnel that should exist in a specific position or work area. The number of personnel in a position can be determined based on work needs, safety regulations and other factors, and is not restricted here.

[0076] By judging whether the number of personnel in the first image is less than the preset number of personnel in the post, it is judged whether there is a vacancy in the target monitoring scene. If the number of personnel in the first image is less than the preset number of personnel in the post, it is determined that there is a vacancy, the cumulative vacancy duration (i.e., the duration of the insufficient number of personnel) is accumulated, and it is detected whether the vacancy duration reaches the preset duration. If the vacancy duration reaches the preset duration, the multimodal large model detection is entered. It can be understood that by judging whether the vacancy duration reaches the preset duration, false alarms caused by accidental factors (such as short-term departure of personnel) can be reduced, making the detection results more accurate and reliable.

[0077] In this embodiment, the method for detecting whether the absence duration has reached the preset duration is not limited, and can be specifically set according to actual needs. For example, in a feasible implementation, when it is first detected that the number of people in the first image is less than the number of people in the post, the starting timestamp start_time can be recorded. In the subsequent frames of monitoring images, if the number of people in the first image is still less than the number of people in the post, the difference between the current time and start_time is calculated, that is, the absence duration is obtained, until the number of people in the first image is greater than or equal to the number of people in the post, the start_time is reset, and then it is detected whether the absence duration has reached the preset duration; for example, in another feasible implementation, the absence duration can be indirectly determined based on a counter and a timer, and a counter counter is set to record the number of consecutive frames of consecutive absences. If the number of consecutive frames of absence is greater than or equal to the preset number of frames, it is determined that the absence duration has reached the preset duration. If the number of consecutive frames of absence is less than the preset number of frames, it is determined that the absence duration has not reached the preset duration. It can be understood that the first implementation method directly calculates based on the timestamp, and has higher detection accuracy than the second implementation method. The second implementation method performs a rough time estimation based on the frame rate, and has a simple implementation process and high detection efficiency compared to the first implementation method. The above two implementation methods are only two feasible implementation methods for detecting whether the absence duration reaches the preset duration. This embodiment does not specifically limit the specific implementation method for detecting whether the absence duration reaches the preset duration.

[0078] Step S30: if the absence duration reaches the preset duration, inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of personnel in the second image of the current frame monitoring image;

[0079] It should be noted that the multimodal target detection model is a model that combines multiple information sources for target detection, and uses information from multiple modes such as images and texts to train and optimize through a deep learning algorithm, thereby achieving high-precision detection of the target. In this embodiment, there is no restriction on the structure of the multimodal target detection model, for example, it can be a multimodal model such as GLIP (Grounded Language-Image Pretraining, language image joint pretraining model), FusionNet (a multimodal fusion model based on deep learning), etc., which can be set according to actual needs.

[0080] It is understandable that when the absence duration reaches the preset duration, the review process of the multimodal target detection model is triggered, that is, the current frame monitoring image is input into the pre-trained multimodal target detection model, and the multimodal target detection model uses information from multiple modalities such as images and texts for comprehensive analysis, and outputs the number of people in the current frame monitoring image, hereinafter referred to as the second image number of people. The multimodal target detection model can combine multiple information sources for personnel detection, and can cope with more complex environmental conditions, such as light changes, occlusions, etc., which improves the robustness of detection and the accuracy of detection results.

[0081] Step S40: If the number of personnel in the second image is less than the number of personnel at the post, it is determined that there are personnel leaving their posts in the target monitoring scene.

[0082] If, after review by the multimodal target detection model, the number of people in the current frame monitoring image is still less than the number of people at the post, it is determined that there are people leaving their posts in the target monitoring scene.

[0083] It should be noted that the personnel absence detection device can be a local device, and the YOLO target detection model and the multimodal target detection model are deployed on the local personnel absence detection device; the personnel absence detection device can also be a cloud device, and the YOLO target detection model and the multimodal target detection model are deployed on the personnel absence detection device on the cloud; the personnel absence detection device can also be a structure combined with a local device and a cloud device, wherein the YOLO target detection model with fast processing speed is deployed on the local personnel absence detection device to obtain preliminary personnel absence detection results more quickly, and the multimodal target detection model with high computing resource consumption is deployed on the cloud personnel absence detection device to obtain more accurate personnel absence detection results, thereby achieving a balance between computing speed and resource consumption.

[0084] In a feasible implementation manner, the step S20: the step of detecting whether the absence duration has reached a preset duration based on the number of personnel in the first image, comprises:

[0085] Step S201, if the number of people in the first image is less than the preset number of people at the post, the target frame number is cumulatively increased by one, and when the target frame number is greater than or equal to the preset frame number, it is determined that the absence duration reaches the preset duration, wherein the target frame number is the number of consecutive frames of the monitoring image in which the number of people in the image is less than the number of people at the post;

[0086] The target frame number indicates the number of frames of surveillance images that continuously detect insufficient personnel. When the number of personnel detected in the first image is less than the preset number of personnel at the post, it indicates that there may be personnel away from their posts, and the target frame number is accumulated and added by one. By accumulating the target frame number, you can intuitively understand how long the shortage of personnel has lasted, which provides a basis for subsequent judgment on whether the absence duration has reached the preset duration.

[0087] It should be noted that the preset number of frames is used to determine whether the absence duration has reached the preset duration threshold, which can be set according to actual needs and is not limited here. Check whether the target number of frames has reached the preset number of frames. If the target number of frames is greater than or equal to the preset number of frames, it means that the absence duration has reached the preset duration. Further, subsequent processing can be performed according to actual needs, such as triggering an alarm, recording an event, etc.

[0088] In a feasible implementation, if the target frame number is less than the preset frame number, it is determined that the absence duration has not reached the preset duration, and more frame images need to be detected for subsequent judgment, that is, return to execute the step of obtaining the current frame monitoring image in the target monitoring scene, and sending the current frame monitoring image to the preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image.

[0089] Step S202: if the number of personnel in the first image is greater than or equal to the number of personnel in the post, the target frame number is cleared, and the process returns to the step of obtaining the current frame monitoring image in the target monitoring scene, and sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of personnel in the first image in the current frame monitoring image;

[0090] If the number of personnel in the current frame monitoring image is greater than or equal to the preset number of personnel at the post, it means that the personnel are in place and the previous personnel leaving the post has ended. The target frame number is cleared to restart the counting.

[0091] In a feasible implementation manner, after the step S30: inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame surveillance image, the step further includes:

[0092] Step S50: If the number of people in the second image is greater than or equal to the number of people in the position, the target frame number is cleared.

[0093] It is understandable that if the number of personnel in the second image is greater than or equal to the preset number of personnel at the post, it means that the number of personnel in the current monitoring area has returned to normal, and the previous possible personnel leaving the post has ended. The target frame number is cleared to restart the counting and evaluation of the personnel leaving the post. By clearing the target frame number in time, it is possible to quickly respond to the situation of personnel returning to their posts, avoiding misjudgment caused by the continuous accumulation of the target frame number, and ensuring that the subsequent personnel leaving the post detection is based on the latest number of personnel, thereby improving the accuracy of the detection.

[0094] Exemplarily, the overall process of personnel leaving their posts can be: S1: Acquire the current frame monitoring image from the image captured by the image sensor. S2: Send the current frame monitoring image to the preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image, and detect whether the number of people in the first image is less than the number of people in the post. If so, go to S3, if not, go to S4. S3: Accumulate and add one to the target frame number, and detect whether the target frame number reaches the preset frame number. If so, determine that the absence time reaches the preset time and execute S5. If not, determine that the absence time does not reach the preset time, and return to execute S1. S4: Clear the target frame number and return to execute S1. S5: Input the preset multimodal target detection model into the current frame monitoring image to obtain the number of people in the second image of the current frame monitoring image, and detect whether the number of people in the second image is less than the number of people in the post. If so, determine that there is a person leaving the post, otherwise, determine that there is no person leaving the post and return to execute S4.

[0095] In a feasible implementation manner, after the step S10: sending the current frame surveillance image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame surveillance image, the step further includes:

[0096] Step S60: if the number of people in the first image is greater than or equal to the number of people in the post, input the current frame surveillance image into a preset face recognition model to obtain first face information of each person in the current frame surveillance image;

[0097] If the number of people in the first image is greater than or equal to the number of people on the job, there may be a situation where the people in the image in the current frame surveillance image are not the staff on the job. At this time, there may be a misjudgment of off-duty detection based on the number of people. Therefore, face recognition is added on the basis of personnel number detection, and the face recognition results are used to determine whether the people in the image are all on-duty staff, so as to obtain the personnel off-duty detection results and improve the accuracy of the detection results.

[0098] Specifically, when the number of people in the first image is greater than or equal to the number of people in the post, the preset face recognition model is used to extract facial information of each person in the current frame surveillance image. It can be understood that the structure and training process of the face recognition model are not restricted here, and can be set according to actual needs. For example, a face recognition model can be constructed based on a convolutional neural network (CNN).

[0099] In this implementation, after obtaining the number of people in the first image of the current frame surveillance image through the YOLO target detection model, if the number of people reaches or exceeds the preset number of people in the positions, the face recognition model is further used to extract the facial feature information of each person in the image, that is, the first face information. By accurately extracting facial features through the face recognition model, on-the-job employees can be further screened and confirmed to improve accuracy.

[0100] Step S70, obtaining second face information preset in the current monitoring scene;

[0101] Obtain the preset facial information of all employees who should be on duty (i.e., post staff) in the current monitoring scene, i.e., the second facial information. The second facial information can be obtained from an employee database, an employee card, or a previous facial recognition record. The specific acquisition method is not limited here and can be set according to actual needs. For example, in one feasible implementation, the preset facial information of all post staff in the current monitoring scene can be queried from the employee database or related system, and this information can be organized into a second facial information list for subsequent comparison and verification.

[0102] Step S80, determining on-duty employees from each of the image personnel based on the first facial information and the second facial information, and determining the number of on-duty employees;

[0103] The extracted first facial information is compared with the preset second facial information to confirm which personnel are on-the-job staff and count the number of on-the-job employees. The specific process can be: traverse each facial feature in the first facial information list; compare each facial feature with the feature in the second facial information list; if the match is successful, mark the person as an on-the-job employee; count the number of on-the-job employees. It can be understood that through facial comparison technology, on-the-job employees can be accurately identified, false positives and false negatives can be avoided, and the number of on-the-job employees can be counted in real time, providing data support for subsequent absence detection.

[0104] Step S90, based on the number of on-duty employees, detects whether the absence duration reaches the preset duration, and executes the step of inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame surveillance image if the absence duration reaches the preset duration.

[0105] After confirming that the number of on-the-job employees is less than the preset number of post personnel, further detection is performed to determine whether the absence duration of the absent personnel reaches the preset duration threshold. The specific detection method can refer to the various implementation methods of step S20, which will not be described in detail here. If the absence duration reaches the preset duration, the current frame monitoring image is input into the preset multimodal target detection model to obtain the number of personnel in the second image in the current frame monitoring image.

[0106] It is understandable that through facial recognition technology, facial comparison technology and absence duration detection, accurate identification of on-the-job employees and real-time monitoring of absences are achieved, providing reliable data support for subsequent absence processing.

[0107] In this embodiment, by combining the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, the YOLO target detection model is used to perform human body detection on the current frame monitoring image, and a large number of video frames can be processed in a short time, thereby ensuring the detection efficiency. The current frame monitoring image is further detected by the multimodal target detection model, and the multimodal model uses more information to improve the accuracy of human body detection. When the multimodal target detection model also detects that the number of personnel is insufficient, it is determined that there are personnel leaving their posts, thereby improving the accuracy of personnel leaving their posts detection.

[0108] In addition, during the personnel leaving their posts detection process, continuous frame detection reduces the number of missed detections by the YOLO target detection model, further improving the accuracy of personnel leaving their posts detection. Moreover, continuous frame detection is performed based on the output of the YOLO model, and does not require additional detection operations, which reduces the complexity of detection and thus avoids affecting the personnel leaving their posts detection.

[0109] Therefore, by combining the high efficiency of the YOLO target detection model and the high accuracy of the multimodal target detection model, the accuracy of personnel leaving their posts is improved while maintaining a high detection efficiency, solving the problem of the inability to balance accuracy and detection efficiency in the current personnel leaving their posts detection process.

[0110] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated hereafter. On this basis, the step S10: sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image includes:

[0111] Step S101, inputting the current frame monitoring image into the YOLO target detection model to obtain the detection frame coordinates of multiple detection frames and the confidence corresponding to each of the detection frames;

[0112] The detection box is a rectangular box used to identify and locate the target object in an image or video. The detection box coordinates refer to the parameters used to describe the position and size of the detection box. The detection box coordinates may include four parameters: the horizontal coordinate of the upper left corner, the vertical coordinate of the upper left corner, the horizontal coordinate of the lower right corner, and the vertical coordinate of the lower right corner. These four parameters together determine the position and size of the detection box in the image. In some feasible implementations, the detection box coordinates may also be parameters such as the center point coordinates, width, and height of the detection box, which are not limited here. The confidence score refers to the predicted probability of the target detection model for whether the detection box contains the target object. The higher the confidence score, the greater the possibility that the model believes that the detection box contains the target object. When the confidence score of the detection box is higher than the threshold, the result of the detection box is considered to be a valid detection result.

[0113] The current frame surveillance image is input into the pre-trained YOLO target detection model, and the model outputs the coordinates of multiple detection boxes and the confidence corresponding to each detection box. It can be understood that through the rapid detection of the YOLO model, multiple detection boxes that may contain people in the current frame surveillance image and their confidence can be efficiently obtained, providing basic data for the subsequent determination of the number of people.

[0114] Step S102, determining a target frame whose confidence is greater than or equal to a preset confidence threshold from each of the detection frames;

[0115] The target box is a detection box that is considered to contain the target object with high confidence after screening and processing. The target box can be a part of the original detection box or the final result after further processing (such as non-maximum suppression).

[0116] The process of determining the target frame can be: traversing all detection frames, obtaining the confidence of each detection frame; comparing the confidence with a preset confidence threshold; screening out detection frames with confidence greater than or equal to the threshold as target frames. It can be understood that by screening out detection frames with higher confidence as target frames for subsequent processing, so as to avoid incorporating falsely detected frames with low confidence into subsequent processing, the number of falsely detected frames can be reduced, and the accuracy and efficiency of subsequent processing can be improved.

[0117] Step S103, traversing each of the target frames in descending order of confidence, determining the currently traversed target frame as the current frame, and calculating the intersection-and-union ratios of each non-current frame and the current frame, wherein the non-current frame is a detection frame in each of the target frames except the current frame;

[0118] Intersection over Union (IoU) is an indicator used to evaluate the degree of overlap between the detection box and the true box in the object detection task. It is defined as the ratio of the intersection area of ​​the detection box and the true box to the union area. The value range of IoU is [0,1]. When the detection box and the true box completely overlap, the IoU is 1. When the detection box and the true box do not overlap at all, the IoU is 0.

[0119] The process of calculating the intersection-and-union ratio of each non-current frame with the current frame may be: traversing all target frames in descending order of confidence; during the traversal process, determining the currently traversed target frame as the current frame; and calculating the intersection-and-union ratio of all other non-current frames with the current frame. It can be understood that by calculating the intersection-and-union ratio, overlapping detection frames can be identified, providing an accurate basis for the subsequent determination of the number of people.

[0120] Step S104: if the IoU is less than or equal to a preset IoU threshold, the object corresponding to the current frame is taken as a person under inspection;

[0121] The intersection-and-union ratio threshold is used to determine whether the current frame overlaps with other frames, thereby determining whether the object corresponding to the current frame is an independent person under inspection. Specifically, for each current frame, its intersection-and-union ratio with all non-current frames is checked. If all intersection-and-union ratios are less than or equal to the preset intersection-and-union ratio threshold, the object corresponding to the current frame is considered to be an independent person under inspection. It can be understood that by judging the intersection-and-union ratio threshold, independent persons under inspection can be accurately identified, avoiding misjudgment of overlapping detection frames as multiple independent persons.

[0122] Step S105, determining the number of the inspected persons, and determining the number of the inspected persons as the number of persons in the first image in the current frame monitoring image.

[0123] Traverse all inspected persons (i.e., objects corresponding to the current frame that meet the intersection and union ratio conditions), count the inspected persons, and use the count result as the number of persons in the first image of the current frame surveillance image. It can be understood that by counting the number of inspected persons, the number of persons in the current frame surveillance image can be accurately obtained, providing key data for subsequent personnel leaving the post detection.

[0124] In this embodiment, through the rapid detection, confidence screening, intersection-over-union calculation and statistics of the number of inspected persons of the YOLO model, the number of persons in the current frame surveillance image can be efficiently obtained, providing accurate basic data for subsequent personnel leaving their posts detection.

[0125] In a feasible implementation manner, before the step S10: sending the current frame surveillance image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame surveillance image, the step further includes:

[0126] Step S01, scaling the current frame monitoring image until the length of the long border of the current frame monitoring image reaches a first preset side length, thereby obtaining a first processed image;

[0127] Adjust the size of the current frame monitoring image so that the length of its long border reaches the preset first preset side length to ensure that the image size input to the YOLO target detection model is uniform, thereby avoiding the detection performance degradation caused by inconsistent image size. The specific process can be: obtain the original size of the current frame monitoring image; compare the relationship between the long side of the original size and the first preset side length; if the long side is smaller than the first preset side length, then scale up the image until the long border length reaches the first preset side length, and obtain the first processed image after scaling; if the long side is larger than the first preset side length, then scale down the image until the long border length reaches the first preset side length, and obtain the first processed image after scaling. It can be understood that through scaling, the image size input to the YOLO model can be ensured to be uniform, thereby improving the detection performance and stability of the model.

[0128] Step S02, adjusting the length of the short border of the first processed image to be consistent with the first preset side length, to obtain a second processed image;

[0129] By filling the short border, the short border length of the first processed image also reaches the first preset side length, thereby obtaining a square or nearly square image to better meet the input requirements of the YOLO model and improve the detection accuracy. Specifically, the short border length of the first processed image is obtained; the relationship between the short border length and the first preset side length is compared; if the short border length is less than the first preset side length, the width of the border to be filled is calculated (usually uniform filling on both sides or up and down); the short border of the first processed image is filled with a filling color such as black, white or the average pixel of the image to obtain the filled second processed image. It can be understood that by filling the short border, an image of uniform size and nearly square can be obtained, which better meets the input requirements of the YOLO model and improves the detection accuracy.

[0130] Step S03, adjusting the image format of the second processed image to a preset YOLO model input format to obtain a third processed image;

[0131] The format of the second processed image is adjusted to the input format required by the YOLO model. Different YOLO versions or different training frameworks may have different requirements for the format of the input image, such as the color channel order (RGB or BGR), image data type (uint8 or float32), etc. Determine the input format required by the YOLO model; check whether the current format of the second processed image is consistent with the input format required by the YOLO model; if not, adjust the format of the second processed image, such as changing the color channel order, converting the image data type, etc., to obtain the third processed image after the format is adjusted. It can be understood that by adjusting the image format, it can be ensured that the image format input to the YOLO model is consistent with the model requirements, avoiding detection errors caused by format mismatch.

[0132] Step S04: convert the third processed image into a tensor image to obtain a fourth processed image, and normalize each pixel value in the fourth processed image to obtain a first target image. Use the first target image as the current frame monitoring image, and execute the step of sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image.

[0133] The third processed image is converted into a tensor image and normalized. The tensor image is a commonly used data representation in deep learning, and the normalization process is to scale the pixel values ​​to a specific range (such as 0 to 1 or -1 to 1) to improve the training efficiency and detection performance of the model. The specific processing process can be: converting the third processed image into a tensor image; calculating the scaling factor required for normalization. In this embodiment, the scaling factor can be the maximum pixel value divided by the maximum value of the target range; applying the scaling factor to each pixel value in the tensor image for normalization, and obtaining the first target image after normalization.

[0134] It can be understood that by converting to a tensor image and performing normalization, it is possible to ensure that the image data input to the YOLO model is standardized, thereby improving the model's training efficiency and detection performance. In addition, normalization can also reduce detection errors caused by differences in image brightness, contrast, etc.

[0135] In this implementation, by steps such as scaling, filling short borders, adjusting the image format, converting to a tensor image, and performing normalization, it is possible to ensure that the image data input to the YOLO target detection model is unified, standardized, and high-quality, thereby improving the detection performance and stability of the model.

[0136] In one feasible implementation, the multimodal target detection model includes a text encoder, an image encoder and a target detection module; the step S30: inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame surveillance image, includes:

[0137] Step S301, obtaining text description information corresponding to the current frame surveillance image, wherein the text description information is used to describe the person in the current frame surveillance image;

[0138] A multimodal target detection model is a machine learning model that can fuse and process information from different modalities (such as images, text, sound, etc.) to perform target detection tasks. The model can utilize the complementarity of data from multiple modalities to improve the accuracy and robustness of target detection. In target detection tasks, traditional single-modal models (such as models based only on images) may be limited by factors such as lighting, occlusion, and changes in perspective. Multimodal target detection models can more comprehensively understand the scene by fusing information from different modalities, thereby detecting targets more accurately. For example, in an autonomous driving scenario, the model can fuse information from cameras (images) and radars (text or numerical data, indicating the distance and speed of objects) to improve the detection accuracy of vehicles and pedestrians.

[0139] Specifically, the multimodal target detection model includes a text encoder, an image encoder and a target detection module. Among them, the text encoder is a machine learning component that converts text data into a numerical feature vector. The text encoder can be a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU) or a Transformer, etc., without limitation. The text encoder learns the potential representation of text data and converts it into a fixed-length feature vector for subsequent classification, clustering or retrieval tasks. The image encoder is a machine learning component that converts image data into a numerical feature vector. The image encoder can be a deep learning model such as a convolutional neural network. The image encoder can learn local features and global structures in the image through structures such as convolutional layers, pooling layers and fully connected layers. Finally, the model converts the image into a fixed-length feature vector. These vectors capture the visual information in the image and can be used for subsequent classification, retrieval or matching tasks. The target detection module is a machine learning component that can identify the location, category and number of target objects in an image. The target detection module can be region candidate-based target detection (such as the R-CNN series), single shot multibox detection (SSD) and YOLO, etc., without limitation. The target detection module can learn the features of the target in the image, locate the target in the image, classify the target, and output the number of targets. The output of the target detection module is usually a set of bounding boxes containing the target location, category and confidence.

[0140] It should be noted that the text description information is used to describe the personnel in the current frame surveillance image, and may include personnel feature information, personnel quantity information and / or personnel distribution information. By obtaining text description information related to the image, additional context information can be provided for subsequent multimodal fusion, thereby improving the accuracy and robustness of target detection.

[0141] There is no restriction on the method of obtaining text description information, which is set according to actual needs. For example, in a feasible implementation, it can be: analyzing the current frame surveillance image based on computer vision technology such as target detection and image segmentation to identify the image personnel in the image; selecting or generating a text description template that matches the current frame surveillance image, for example, if multiple persons are detected, a template such as "There are X persons in the image" can be used; filling the identified key elements and attributes into the text template to generate the final text description information.

[0142] Step S302, inputting the current frame monitoring image into the image encoder to obtain an image feature vector, and inputting the text description information into the text encoder to obtain a text feature vector;

[0143] The image encoder converts the current frame monitoring image into an image feature vector, while the text encoder converts the text description information into a text feature vector. By extracting the feature vectors of the image and text respectively, the information of the two modalities can be fully utilized, providing a basis for subsequent multimodal fusion.

[0144] Step S303, fusing the image feature vector and the text feature vector to obtain a depth feature vector;

[0145] The image feature vector and the text feature vector are fused to obtain a deep feature vector containing information from both modalities. The specific feature vector fusion method can be concatenation, dot product, weighted sum, etc., which is not limited here. By fusing the feature vectors of the image and text, a deep feature vector containing information from both modalities can be obtained, which helps improve the accuracy and generalization ability of target detection.

[0146] Step S304: input the depth feature vector into the target detection module to obtain the number of people in the second image in the current frame surveillance image.

[0147] The deep feature vector is input into the target detection module to obtain the number of people in the second image in the current frame surveillance image. By using multimodal information composed of images and texts for target detection, a more accurate estimation of the number of people can be obtained. In addition, due to the fusion of information from multiple modalities, the model is more robust to changes in complex scenes and lighting conditions.

[0148] In a feasible implementation manner, before the step S30: inputting the current frame surveillance image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame surveillance image, the method includes:

[0149] Step S05, converting the current frame monitoring image into a tensor image to obtain a fifth processed image;

[0150] The current frame monitoring image is converted from a traditional image format (such as JPEG, PNG, etc.) to a tensor format, and the converted image is called the fifth processed image for distinction. The specific conversion process can be: read the current frame monitoring image and obtain its pixel data; create a tensor of a corresponding shape according to the width, height and number of channels of the image (such as 3 channels for RGB images); fill the pixel data of the image into the tensor to complete the conversion.

[0151] Step S06, scaling the fifth processed image until the length of the short border of the fifth processed image reaches a second preset side length, thereby obtaining a sixth processed image;

[0152] The converted tensor image is scaled to obtain a sixth processed image to ensure that the image size meets the input requirements of the preset multimodal target detection model. The scaling process usually involves adjusting the width and height of the image to keep the aspect ratio of the image unchanged and make the short side of the image reach a specified side length, that is, the second preset side length. The specific scaling process is not described here.

[0153] Step S07, adjusting the image format of the sixth processed image to a preset multimodal model input format to obtain a seventh processed image;

[0154] The image format of the sixth processed image is adjusted to the input format required by the preset multimodal target detection model to obtain the seventh processed image. It should be noted that different deep learning models may require different input formats, such as specific data types, channel orders, etc. The specific processing process is not described here.

[0155] Step S08, normalize the pixel values ​​of each pixel in the seventh processed image to obtain a second target image, use the second target image as the current frame monitoring image, and execute the step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the second image number of people in the current frame monitoring image.

[0156] The resized images are standardized, that is, the pixel values ​​of the images are scaled to a specific range (usually between 0 and 1 or between -1 and 1) to eliminate the brightness and contrast differences between different images, thereby improving the generalization ability of the model.

[0157] It can be understood that through the image preprocessing operation, the current frame monitoring image is converted into a format and size that meets the input requirements of the preset multimodal target detection model, while eliminating the brightness differences and contrast differences between images, providing high-quality input data for subsequent model detection.

[0158] For example, in order to help understand the implementation process of the personnel leaving the post detection method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 2 , Figure 2 A brief flow chart of a method for detecting personnel leaving their posts is provided, which mainly includes five main steps: YOLOv5 human detection, number of people judgment, continuous frame number judgment, GLIP human detection, and comprehensive judgment.

[0159] Step 1: Get the image in the video stream and the input fixed number of personnel P_num (the number of personnel when there is no off-duty situation), pre-process the image, use the YOLOv5 target detection algorithm to detect the human body in the image, and obtain the number of personnel. Step 2: If the number of personnel detected in step 1 is less than the fixed number of personnel, the number of consecutive frames less than the fixed number of people is increased by 1, otherwise the number of consecutive frames less than the fixed number of people is cleared to 0, and the next frame image is directly detected. Step 3: Determine whether the number of consecutive frames less than the fixed number of people is greater than the set threshold. If it is greater, use the GLIP algorithm to detect the human body in the current frame image, otherwise continue to detect the next frame image. Step 4: When the number of consecutive frames less than the fixed number of people is greater than the set threshold, use GLIP to detect the human body in the current frame image and obtain the number of personnel. Step 5: If the number of personnel detected in step 4 is less than the fixed number of personnel, it is judged that there is off-duty situation and the number of consecutive frames less than the fixed number of people is cleared to 0, otherwise the number of consecutive frames less than the fixed number of people is cleared to 0.

[0160] In this embodiment, in step 1, the image is first preprocessed based on the requirements of the YOLOv5 algorithm. The process is as follows: Figure 3As shown, the specific operations are as follows: Process the long side of the image: scale the long side of the input image to 640 while keeping the aspect ratio unchanged. Process the short side of the image: for the image with the long side adjusted to 640, fill the short border with (114,114,114) pixel values ​​to make the short side length 640. Convert the input format: convert the image from HWC format to CHW format, where C represents the color channel, H represents the image height, and W represents the image width; adjust the image color channel from BGR to RGB. Change the image type: convert the image from an array (numpy) to a tensor (Tensor) to meet the input format requirements of the network built based on the pytorch framework. Normalize pixel values: adjust all pixel values ​​of the image to the range of (0,1). After the image preprocessing is completed, the adjusted image is sent to the YOLOv5 network, and YOLOv5 detects the objects in the image, and then screens the detection results, returning only the results related to the human body. The specific steps are as follows: Multi-frame parallel detection: In order to speed up the detection speed, the multi-frame image parallel detection method is adopted, and the preprocessed image is stacked in a new dimension to obtain a tensor Images of size (N, C, H, W), where N represents the number of image frames, C represents the image channel (RGB image has three channels), H is the image length, and W is the image width. The tensor Images is sent to the YOLOv5 network for prediction to obtain the result outputs, which contains the detection box coordinates and confidence. The detection box coordinates are in the form of (x1, y1, x2, y2), which represent the coordinate values ​​of the upper left corner and the lower right corner of the detection box respectively, and the confidence indicates the credibility of the object framed by the detection box belonging to the corresponding category. Detection result processing: Clean the outputs through non-maximum suppression (NMS) to remove redundant boxes of the same target. The process is to first discard the prediction boxes with a confidence level less than 0.25 in outputs, and then sort the remaining prediction boxes from large to small according to the confidence level to obtain the prediction box set H. Starting from the box h with the highest confidence in H, calculate the intersection-over-union (IOU) of all other boxes, and remove the boxes with an intersection-over-union ratio greater than 0.45 with h from H. Repeat the above operation for other boxes in H, and finally obtain the cleaned detection result outputs2. The calculation formula of IOU is:

[0161]

[0162] Among them, the frame intersection and frame union parts can be as follows Figure 4As shown in the dark part. Detection result screening: This embodiment only involves the human body, so the human body result person_output needs to be screened out from outputs2. person_output includes the coordinates and confidence of each human body frame in each image. By counting the human body frames in each frame image, the number of people detected in each frame image is obtained and recorded in the list person_num (each number in person_num represents the number of people detected in the corresponding frame image). In step 2, based on the person_num list obtained in step 1, it is determined whether there is a situation where the corresponding frame image has a person leaving his post, where F_num represents the number of consecutive frames where the number of people detected is less than P_num. The larger F_num is, the greater the probability that the algorithm believes that there is a person leaving his post. The specific operation is to traverse each value n in person_num and compare it with P_num. If n is less than P_num, it is preliminarily determined that there may be a person leaving his post in the frame image, and the number of consecutive frames less than the fixed number of people F_num is increased by 1, otherwise F_num is cleared to 0 and the result of the next frame is directly determined. The reason for this step is that after many experiments, it was found that YOLOv5 is easily affected by occlusion factors and may miss detection. However, for continuous frames in the video, there is a high probability that several frames can detect all the human bodies in the image normally. Based on the consideration of this feature of YOLOv5 and the result of counting the number of people in multiple consecutive frames in step 1, when the number of people in person_num is not less than P_num, it is assumed that the result of the number of people less than P_num is caused by factors such as occlusion, and the previous accumulated results of F_num are also determined to be missed detection, and F_num is cleared to 0. Step 3 is a logical judgment step. When F_num is added by 1 in step 2, it is determined whether GLIP needs to be used for further detection based on the situation of F_num. Assume that in step 2, when traversing to person_num[i], the F_num addition operation occurs, and the image corresponding to person_num[i] is image. Step 3 will determine whether the number of consecutive frames less than P_num, that is, F_num is greater than the threshold thresh. If so, the algorithm believes that there is a high probability of leaving the post, and the image frame image is sent to the GLIP network for human detection, where thresh is set to half of the number of image frames input in step 1. For example, 10 frames of images are extracted from the video stream for leaving the post detection. If there are more than 5 consecutive frames with the number of human bodies less than P_num, it is preliminarily determined that there is a high probability of personnel leaving the post, and the GLIP model will confirm the detection of the current frame image. If it is determined according to step 3 that GLIP needs to perform confirmation detection, go to step 4. GLIP is a multimodal target detection model that combines target detection and phrase localization tasks for pre-training. This unification makes the training data more sufficient. At the same time, the feature representation learned by the model has rich semantic information, which helps to improve the accuracy of target detection.Based on these advantages of GLIP, through experiments in real scenes, it can be found that GLIP is better than YOLOv5 in detecting human bodies under occlusion. Therefore, this embodiment uses GLIP to perform secondary confirmation on the off-duty result judged by YOLOv5. Since GLIP is only called for confirmation detection when YOLOv5 detects off-duty, the detection time is not greatly increased while improving the detection effect. Different from YOLOv5, the input of GLIP includes two modes, image and text. The text specifies which type of target needs to be detected in the image. If the human body needs to be detected in the present invention, the input text is "person". The specific process is to first perform image preprocessing on the image in step 3, and then send the processed image and the set text text as "person" into the GLIP network for detection. Image preprocessing is as follows. Figure 5 As shown, it mainly includes the following operations: Change the image type: convert the image from an array (numpy) type to a Tensor to meet the input format requirements of the network built on the pytorch framework. Process the image size: scale the short side of the input image to 800 while keeping the aspect ratio unchanged. Convert the input format: convert the image from HWC format to CHW format, where C represents the color channel, H represents the image height, and W represents the image width. Standard pixel value: standardize all pixel values ​​of the image, that is, subtract the mean and divide by the variance, which conforms to the standard normal distribution. The mean and variance values ​​are [103.530, 116.280, 123.675] and [57.375, 57.120, 58.395], respectively. After preprocessing, the image is sent to the GLIP model together with the text. Please refer to Figure 6, the GLIP model is generally composed of two parts, a text encoder and an image encoder. The text encoder text_encoder adopts the Bert model, which is responsible for processing text and generating a text feature vector text_feature. The image encoder image_encoder is a ViT-L / 14 model, which is used to encode images and extract image feature vectors image_feature. After the deep fusion operation, text_feature and image_feature are used to predict the human body in the image and generate the result output_GLIP, including the detection quantity position coordinates and confidence. Among them, the coordinate form is (x1, y1, x2, y2), which respectively represent the coordinate values ​​of the upper left corner and the lower right corner of the detection frame. The confidence indicates the degree of credibility that the object framed by the detection frame belongs to the corresponding category. In this embodiment, 0.5 is used as the confidence threshold, that is, only the detection frame with a confidence value greater than 0.5 is returned, and the detection frame with a confidence value less than 0.5 is automatically filtered out. Step 5 is to judge the result obtained in step 4. According to output_GLIP, the number of people glip_num in the image image detected by GLIP can be obtained. If glip_num is less than P_num, it can be considered that there is off-duty behavior in the video stream passed in step 1, an alarm is issued, and F_num is cleared to 0. This step can alleviate the time loss caused by continuous calling GLIP in subsequent frames after off-duty is detected; if glip_num is not less than P_num, it is considered that YOLOv5's judgment is inaccurate, and F_num is cleared to 0.

[0163] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the present applicant's method for detecting off-duty personnel. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0164] The present application also provides a personnel leaving post detection device. The personnel leaving post detection device provided by the present application adopts the personnel leaving post detection method in the above embodiment, which can solve the technical problem that the accuracy and detection efficiency cannot be balanced in the current personnel leaving post detection process. Compared with the prior art, the beneficial effects of the personnel leaving post detection device provided by the present application are the same as the beneficial effects of the personnel leaving post detection method provided by the above embodiment, and the other technical features in the personnel leaving post detection device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0165] The present application provides a personnel absence detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the personnel absence detection method in the above-mentioned embodiment.

[0166] Reference below Figure 7 , which shows a schematic diagram of the structure of a personnel absence detection device suitable for implementing the embodiment of the present application. The personnel absence detection device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The personnel absence detection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0167] like Figure 7 As shown, the personnel absence detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the personnel absence detection device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the personnel absence detection device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a personnel absence detection device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0168] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0169] The personnel leaving post detection device provided by the present application adopts the personnel leaving post detection method in the above embodiment, which can solve the technical problem that the accuracy and detection efficiency cannot be balanced in the current personnel leaving post detection process. Compared with the prior art, the beneficial effects of the personnel leaving post detection device provided by the present application are the same as the beneficial effects of the personnel leaving post detection method provided by the above embodiment, and the other technical features in the personnel leaving post detection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0170] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0171] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0172] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the personnel absence detection method in the above-mentioned embodiment.

[0173] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0174] The computer-readable storage medium may be included in the personnel absence detection device; or may exist independently without being assembled into the personnel absence detection device.

[0175] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the personnel absence detection device, the personnel absence detection device executes the contents of the above embodiments.

[0176] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0179] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned personnel off-duty detection method, and can solve the technical problem that the accuracy and detection efficiency cannot be balanced in the current personnel off-duty detection process. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the personnel off-duty detection method provided by the above-mentioned embodiment, and will not be repeated here.

[0180] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned personnel absence detection method when executed by a processor.

[0181] The computer program product provided by the present application can solve the technical problem that the accuracy and detection efficiency cannot be balanced in the current personnel off-duty detection process. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the personnel off-duty detection method provided by the above embodiment, which will not be described in detail here.

[0182] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for detecting when a person is away from his post, characterized in that: The personnel leaving the post detection method comprises: Acquire a current frame monitoring image in a target monitoring scene, and send the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image; Based on the number of personnel in the first image, detecting whether the absence duration has reached a preset duration, wherein the absence duration is a continuous duration during which the number of personnel in the first image is less than the number of personnel in the post; If the absence duration reaches the preset duration, the current frame monitoring image is input into a preset multimodal target detection model to obtain the number of personnel in the second image of the current frame monitoring image; If the number of personnel in the second image is less than the number of personnel at the post, it is determined that there are personnel leaving their posts in the target monitoring scene.

2. The method for detecting personnel leaving their post as claimed in claim 1, characterized in that: The step of detecting whether the absence duration has reached a preset duration based on the number of personnel in the first image comprises: If the number of people in the first image is less than the preset number of people at the post, the target number of frames is cumulatively added by one, and when the target number of frames is greater than or equal to the preset number of frames, it is determined that the absence duration reaches the preset duration, wherein the target number of frames is the number of consecutive frames of the surveillance image in which the number of people in the image is less than the number of people at the post; If the number of people in the first image is greater than or equal to the number of people in the position, the target frame number is cleared, and the process returns to the step of obtaining the current frame monitoring image under the target monitoring scene, and sending the current frame monitoring image to the preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image.

3. The method for detecting personnel leaving their post as claimed in claim 2, characterized in that: After the step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame monitoring image, the method further includes: If the number of people in the second image is greater than or equal to the number of people in the position, the target number of frames is cleared.

4. The method for detecting personnel leaving their post as claimed in claim 1, characterized in that: The step of sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image includes: Input the current frame monitoring image into the YOLO target detection model to obtain the detection frame coordinates of multiple detection frames and the confidence corresponding to each of the detection frames; Determine, from each of the detection frames, a target frame whose confidence is greater than or equal to a preset confidence threshold; Traversing each of the target frames in descending order of confidence, determining the currently traversed target frame as the current frame, and calculating the intersection and union ratio of each non-current frame with the current frame, wherein the non-current frame is a detection frame in each of the target frames except the current frame; If the intersection-over-union ratio is less than or equal to a preset intersection-over-union ratio threshold, the object corresponding to the current frame is regarded as the inspected person; The number of the inspected persons is determined, and the number of the inspected persons is determined as the number of persons in the first image in the current frame monitoring image.

5. The method for detecting personnel leaving their post as claimed in claim 1, characterized in that: Before the step of sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image of the current frame monitoring image, the method further includes: Scaling the current frame monitoring image until the length of the long border of the current frame monitoring image reaches a first preset side length to obtain a first processed image; Adjusting the length of the short border of the first processed image to be consistent with the first preset side length to obtain a second processed image; Adjusting the image format of the second processed image to a preset YOLO model input format to obtain a third processed image; The third processed image is converted into a tensor image to obtain a fourth processed image, and each pixel value in the fourth processed image is normalized to obtain a first target image. The first target image is used as the current frame monitoring image, and the step of sending the current frame monitoring image to a preset YOLO target detection model to obtain the number of people in the first image in the current frame monitoring image is executed.

6. The method for detecting personnel leaving their post as claimed in claim 1, characterized in that: The multimodal object detection model includes a text encoder, an image encoder and an object detection module; The step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame monitoring image includes: Obtaining text description information corresponding to the current frame surveillance image, wherein the text description information is used to describe the person in the current frame surveillance image; Inputting the current frame monitoring image into the image encoder to obtain an image feature vector, and inputting the text description information into the text encoder to obtain a text feature vector; Fusion of the image feature vector and the text feature vector to obtain a depth feature vector; The depth feature vector is input into the target detection module to obtain the number of people in the second image in the current frame monitoring image.

7. The method for detecting personnel leaving their post according to any one of claims 1 to 6, characterized in that: Before the step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image of the current frame monitoring image, the method includes: Converting the current frame monitoring image into a tensor image to obtain a fifth processed image; Scaling the fifth processed image until the length of the short border of the fifth processed image reaches a second preset side length, thereby obtaining a sixth processed image; Adjusting the image format of the sixth processed image to a preset multimodal model input format to obtain a seventh processed image; The pixel values ​​of each pixel in the seventh processed image are standardized to obtain a second target image, the second target image is used as the current frame monitoring image, and the step of inputting the current frame monitoring image into a preset multimodal target detection model to obtain the number of people in the second image in the current frame monitoring image is executed.

8. A personnel leaving the post detection device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the personnel leaving their post detection method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the personnel leaving the post detection method as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the personnel absence detection method according to any one of claims 1 to 7 are implemented.

Images