Method and device for identifying idling behavior of security officer, and electronic equipment

Abstract

The invention discloses a method and a device for identifying idling behaviors of security officers and electronic equipment, and relates to the field of artificial intelligence and big data, and the method comprises the steps: obtaining a to-be-identified image frame set; performing target identification on the image frame set, and performing tracking identification on a security officer in the image frame based on a target tracking algorithm under the condition that the security officer is on duty in the image frame to obtain a second identification result; performing target detection on the image frame set, and performing association identification and key point identification on the security officer and the handheld terminal under the condition that a target detection result indicates that the handheld terminal exists in the image frame to obtain a third identification result; and generating an idling behavior recognition result based on the first recognition result, the second recognition result and the third recognition result. According to the method and the device, the technical problem that the identification result is low in accuracy due to the fact that the mode of identifying the idling behavior of the security officer based on target detection depends on a single-frame image in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and big data, and in particular, to a method and device for identifying the slacking behavior of a security officer, and an electronic device. Background Art

[0002] In the modern fields of monitoring and safety management, identifying the slacking behavior of security officers has become a key link in improving work efficiency and safety levels. The slacking behavior of security officers will directly affect the response speed and efficiency of security officers to emergencies, which may have an adverse impact on personnel safety, property safety, etc. Therefore, real-time and accurate monitoring and identification of the slacking behavior of security officers are crucial for maintaining emergency response, park management, production safety and other fields.

[0003] In the related art, monitoring and identifying slacking behavior mainly rely on video monitoring and computer vision technologies. Specifically, whether a security officer has slacking behavior can be identified through object detection. A single object detection technology usually relies on a single-frame image, which may lead to false alarms or missed detections, because some normal behaviors (such as checking the time, answering an emergency call, etc.) may be misidentified as slacking behavior, resulting in misjudgment and low accuracy of the identification result.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present invention provide a method and device for identifying the slacking behavior of a security officer, and an electronic device, so as to at least solve the technical problem in the related art that the method for identifying the slacking behavior of a security officer based on object detection relies on a single-frame image and has low accuracy of the identification result.

[0006] According to one aspect of the embodiments of the present invention, a method for identifying the slacking behavior of a security officer is provided, including: obtaining video data, and preprocessing the video data to obtain a set of image frames to be identified; performing target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the security officer is on duty; in the case that the first recognition result indicates that the security officer in the image frame is on duty, performing tracking recognition on the security officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the security officer has the behavior of sleeping on duty; performing target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; in the case that the target detection result indicates that there is a handheld terminal in the image frame, performing association recognition and key point recognition on the security officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the security officer has the behavior of using the handheld terminal; generating a slacking behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result.

[0007] Further, performing target recognition on the image frames in the set of image frames to obtain a first recognition result includes: performing target recognition on the image frames in the set of image frames based on preset recognition features to identify whether there is a security officer in a preset detection area; in the case that there is no security officer in the preset detection area of the target image frame, identifying the off-duty behavior of the security officer based on adjacent image frames, and in the case that it is determined that the security officer has an off-duty behavior, generating an off-duty recognition result for the security officer with the off-duty behavior; or, in the case that there is a security officer in all the image frames in the set of image frames, determining that the security officer has no off-duty behavior and generating an on-duty recognition result for the security officer without the off-duty behavior; generating the first recognition result based on the off-duty recognition result or the on-duty recognition result.

[0008] Further, the step of identifying the off-duty behavior of the security officer based on adjacent image frames in the case that there is no security officer in the preset detection area of the target image frame includes: in the case that there is no security officer in the preset detection area of the target image frame, obtaining the adjacent image frames corresponding to the target image frame, and detecting whether there is a security officer in the preset detection area of G of the adjacent image frames, where G is a positive integer; in the case that there is no security officer in all G of the adjacent image frames, determining that the security officer has a suspected off-duty behavior, and calculating the suspected off-duty duration based on the time stamp corresponding to the image frame; comparing the suspected off-duty duration with a preset off-duty duration threshold to obtain a first comparison result; in the case that the first comparison result indicates that the suspected off-duty duration is greater than the off-duty duration threshold, determining that the security officer has an off-duty behavior.

[0009] Further, when the first recognition result indicates that the safety officer is on duty in the image frame, the steps of performing tracking recognition on the safety officer in the image frame based on the target tracking algorithm to obtain a second recognition result include: when the first recognition result indicates that the safety officer is on duty in the image frame, performing pose analysis on the safety officer in the target image frame to determine whether the safety officer is in a sleeping state, where the pose analysis at least includes: head tilt degree analysis, eye closure degree analysis, body posture state analysis; when the safety officer is in a sleeping state, performing tracking recognition on the safety officer in the adjacent image frames corresponding to the target image frame, and when the safety officers in K adjacent image frames are all in a sleeping state, calculating the sleeping duration of the safety officer, where K is a positive integer; comparing the sleeping duration with a preset sleeping duration threshold to obtain a second comparison result, and when the second comparison result indicates that the sleeping duration is greater than the sleeping duration threshold, determining that the safety officer has a behavior of sleeping on duty, generating a sleeping behavior recognition result for the safety officer with the behavior of sleeping on duty, and obtaining a second recognition result based on the sleeping behavior recognition result.

[0010] Further, when the target detection result indicates that there is a handheld terminal in the image frame, the steps of performing associated recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result include: when there is a handheld terminal in the image frame, using the Hungarian algorithm to perform associated recognition on the safety officer and the handheld terminal in the target image frame with the handheld terminal to obtain a first associated recognition result, and the associated recognition includes recognizing the relative position and relative distance between the safety officer and the handheld terminal; when the first associated recognition result indicates that there is an association between the safety officer and the handheld terminal, using the Hungarian algorithm to perform associated recognition on the handheld terminal and the hand key points of the safety officer to obtain a second associated recognition result; when the second associated recognition result indicates that there is an association between the handheld terminal and the hand key points of the safety officer, recognizing the relative relationship between the target key points of the handheld terminal and the safety officer to obtain a third recognition result, where the target key points include at least one of the following: ear key points, eye key points.

[0011] Further, the steps of identifying the relative relationship between the handheld terminal and the target key points of the security officer include: obtaining the largest acute angle in the triangular relationship formed by the eye key points of the security officer and the handheld terminal; comparing the angle value of the largest acute angle with a preset angle threshold to obtain a third comparison result; in the case where the third comparison result indicates that the angle value of the largest acute angle is less than the preset angle threshold, determining that the security officer has a first suspected terminal usage behavior, and obtaining the target image frame in which the security officer has the first suspected terminal usage behavior; based on the adjacent image frames corresponding to the target image frame, tracking and identifying the security officer with the first suspected terminal usage behavior, and calculating the first terminal usage duration of the security officer when the security officers in M adjacent image frames all have the first suspected terminal usage behavior, where M is a positive integer; in the case where the first terminal usage duration is greater than a preset terminal usage duration threshold, determining that the security officer has the behavior of using the handheld terminal for entertainment.

[0012] Further, the steps of identifying the relative relationship between the handheld terminal and the target key points of the security officer further include: obtaining the distance value between the ear key points of the security officer and the handheld terminal; comparing the distance value with a preset distance threshold to obtain a fourth comparison result; in the case where the fourth comparison result indicates that the distance value is less than the preset distance threshold, determining that the security officer has a second suspected terminal usage behavior, and obtaining the target image frame in which the security officer has the second suspected terminal usage behavior; based on the adjacent image frames corresponding to the target image frame, tracking and identifying the security officer with the second suspected terminal usage behavior, and calculating the second terminal usage duration of the security officer when the security officers in N adjacent image frames all have the second suspected terminal usage behavior, where N is a positive integer; in the case where the second terminal usage duration is greater than a preset terminal usage duration threshold, determining that the security officer has the behavior of using the handheld terminal to make a call.

[0013] Further, the steps of generating a dereliction of duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result further include: in the case where the third recognition result indicates that the security officer has a handheld terminal usage behavior during on-duty, correcting the second recognition result based on the third recognition result to obtain the corrected second recognition result; generating the dereliction of duty behavior recognition result based on the first recognition result, the corrected second recognition result, and the third recognition result.

[0014] According to another aspect of the embodiments of the present invention, there is also provided a recognition device for the slacking behavior of a safety officer, including: an acquisition unit, configured to acquire video data and preprocess the video data to obtain a set of image frames to be recognized; a first recognition unit, configured to perform target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the safety officer is on duty; a second recognition unit, configured to, when the first recognition result indicates that the safety officer in the image frame is on duty, perform tracking recognition on the safety officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has a behavior of sleeping on duty; a detection unit, configured to perform target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; a third recognition unit, configured to, when the target detection result indicates that there is a handheld terminal in the image frame, perform association recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a behavior of using the handheld terminal; a generation unit, configured to generate a slacking behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result.

[0015] Further, the first recognition unit includes: a first recognition module, configured to perform target recognition on the image frames in the set of image frames based on preset recognition features to recognize whether there is a safety officer in a preset detection area; a second recognition module, configured to, when there is no safety officer in the preset detection area of the target image frame, recognize the off-duty behavior of the safety officer based on adjacent image frames, and generate an off-duty recognition result for the safety officer with the off-duty behavior when it is determined that the safety officer has the off-duty behavior; a third recognition module, configured to determine that the safety officer has no off-duty behavior and generate an on-duty recognition result for the safety officer without the off-duty behavior when all the image frames in the set of image frames have a safety officer; a first generation module, configured to generate the first recognition result based on the off-duty recognition result or the on-duty recognition result.

[0016] Further, the second recognition module includes: a first acquisition sub-module, configured to acquire an adjacent image frame corresponding to the target image frame when a safety officer does not exist in a preset detection area of the target image frame, and detect whether a safety officer exists in the preset detection areas of G adjacent image frames, where G is a positive integer; a first determination sub-module, configured to determine that the safety officer has a suspected off-duty behavior and calculate a suspected off-duty duration based on the time stamp corresponding to the image frame when no safety officer exists in all of the G adjacent image frames; a first comparison sub-module, configured to compare the suspected off-duty duration with a preset off-duty duration threshold to obtain a first comparison result; and a second determination sub-module, configured to determine that the safety officer has an off-duty behavior when the first comparison result indicates that the suspected off-duty duration is greater than the off-duty duration threshold.

[0017] Further, the second recognition unit includes: a first analysis module, configured to perform pose analysis on the safety officer in the target image frame to determine whether the safety officer is in a sleeping state when the first recognition result indicates that the safety officer is on duty in the image frame, where the pose analysis at least includes: head tilt degree analysis, eye closure degree analysis, and body posture state analysis; a fourth recognition module, configured to perform tracking recognition on the safety officer in an adjacent image frame corresponding to the target image frame when the safety officer is in a sleeping state, and calculate the sleeping duration of the safety officer when the safety officers in K adjacent image frames are all in a sleeping state, where K is a positive integer; a second comparison module, configured to compare the sleeping duration with a preset sleeping duration threshold to obtain a second comparison result, and determine that the safety officer has a behavior of sleeping on duty, generate a sleeping behavior recognition result for the safety officer having the behavior of sleeping on duty, and obtain a second recognition result based on the sleeping behavior recognition result.

[0018] Further, the third recognition unit includes: a fifth recognition module, configured to, when a handheld terminal exists in the image frame, use the Hungarian algorithm to perform association recognition on the safety officer and the handheld terminal in the target image frame where the handheld terminal exists, so as to obtain a first association recognition result, and the association recognition includes recognizing the relative position and relative distance between the safety officer and the handheld terminal; a sixth recognition module, configured to, when the first association recognition result indicates an association between the safety officer and the handheld terminal, use the Hungarian algorithm to perform association recognition on the handheld terminal and the hand key points of the safety officer, so as to obtain a second association recognition result; a seventh recognition module, configured to, when the second association recognition result indicates an association between the handheld terminal and the hand key points of the safety officer, recognize the relative relationship between the handheld terminal and the target key points of the safety officer, so as to obtain a third recognition result, where the target key points include at least one of the following: ear key points and eye key points.

[0019] Further, the seventh recognition module includes: a second acquisition sub-module, configured to acquire the largest acute angle in the triangular relationship formed by the eye key points of the safety officer and the handheld terminal; a second comparison sub-module, configured to compare the angle value of the largest acute angle with a preset angle threshold to obtain a third comparison result; a third determination sub-module, configured to, when the third comparison result indicates that the angle value of the largest acute angle is less than the preset angle threshold, determine that the safety officer has a first suspected terminal usage behavior, and acquire the target image frame in which the safety officer has the first suspected terminal usage behavior; a first calculation sub-module, configured to perform tracking recognition on the safety officer with the first suspected terminal usage behavior based on the adjacent image frames corresponding to the target image frame, and calculate the first terminal usage duration of the safety officer when the safety officers in all M adjacent image frames have the first suspected terminal usage behavior, where M is a positive integer; a fourth determination sub-module, configured to, when the first terminal usage duration is greater than a preset terminal usage duration threshold, determine that the safety officer has the behavior of using the handheld terminal for entertainment.

[0020] Further, the seventh recognition module further includes: a third acquisition sub-module, configured to acquire the distance value between the ear key points of the safety officer and the handheld terminal; a third comparison sub-module, configured to compare the distance value with a preset distance threshold to obtain a fourth comparison result; a fifth determination sub-module, configured to determine that the safety officer has a second suspected terminal usage behavior and acquire a target image frame of the safety officer having the second suspected terminal usage behavior when the fourth comparison result indicates that the distance value is less than the preset distance threshold; a second calculation sub-module, configured to perform tracking and recognition on the safety officer having the second suspected terminal usage behavior based on the adjacent image frames corresponding to the target image frame, and calculate the second terminal usage duration of the safety officer when the safety officers in N adjacent image frames all have the second suspected terminal usage behavior, where N is a positive integer; a sixth determination sub-module, configured to determine that the safety officer has the behavior of making a call using the handheld terminal when the second terminal usage duration is greater than a preset terminal usage duration threshold.

[0021] Further, the generating unit includes: a first correction module, configured to correct the second recognition result based on the third recognition result to obtain the corrected second recognition result when the third recognition result indicates that the safety officer has a handheld terminal usage behavior while on duty; a second generating module, configured to generate the absent-duty behavior recognition result based on the first recognition result, the corrected second recognition result, and the third recognition result.

[0022] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory, where the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for recognizing the absent-duty behavior of a safety officer according to any one of the above.

[0023] In this application, the following steps are used to identify the behavior of a safety officer neglecting their duties: Obtain video data, and preprocess the video data to obtain a set of image frames to be recognized; Perform object recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the safety officer is on duty; When the first recognition result indicates that the safety officer in the image frame is on duty, perform tracking recognition on the safety officer in the image frame based on the object tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has the behavior of sleeping on duty; Perform object detection on the set of image frames to obtain an object detection result, where the object detection result is used to indicate whether there is a handheld terminal in the image frame; When the object detection result indicates that there is a handheld terminal in the image frame, perform association recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has the behavior of using a handheld terminal; Generate a recognition result for the behavior of neglecting duties based on the first recognition result, the second recognition result, and the third recognition result.

[0024] In this application, through multiple recognition technologies such as object detection, tracking recognition, association recognition, and key point recognition, object detection and association recognition are performed on multiple image frames in the set of image frames, reducing false alarms caused by single recognition methods and single recognition content, achieving fast and accurate recognition of the behavior of a safety officer neglecting their duties, improving the accuracy of the recognition result for the behavior of neglecting duties, and thus solving the technical problem in the related art that the method of identifying the behavior of a safety officer neglecting their duties based on object detection relies on single-frame images and has a low accuracy of the recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for identifying the behavior of a safety officer neglecting their duties is shown;

[0027] Figure 2 It is a flowchart of an optional method for identifying the behavior of a safety officer neglecting their duties according to an embodiment of the present invention;

[0028] Figure 3 It is an architecture diagram of an optional system for identifying the behavior of a safety officer neglecting their duties according to an embodiment of the present invention;

[0029] Figure 4 It is a schematic diagram of an optional recognition process for the behavior of a safety officer neglecting their duties according to an embodiment of the present invention;

[0030] Figure 5It is a schematic diagram of an optional method for identifying the slacking behavior of a safety officer based on eye key points according to an embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of an optional method for identifying the slacking behavior of a safety officer based on ear key points according to an embodiment of the present invention;

[0032] Figure 7 It is a schematic diagram of an optional identification device for the slacking behavior of a safety officer according to an embodiment of the present invention;

[0033] Figure 8 It is a hardware structure block diagram of an electronic device (or mobile device) that optionally executes the method for identifying the slacking behavior of a safety officer according to an embodiment of the present invention. Detailed implementation manners

[0034] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the method and device for identifying the slacking behavior of a safety officer in the present application can be used in the field of artificial intelligence and big data for identifying the slacking behavior of a safety officer, and can also be used in any field other than the field of artificial intelligence and big data for identifying the slacking behavior of a safety officer. The application field of the method and device for identifying the slacking behavior of a safety officer in the present application is not limited.

[0037] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Moreover, for the processing of relevant data such as collection, storage, use, processing, transmission, provision, disclosure, and application, all comply with relevant laws, regulations, and standards, adopt necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, there is an interface between this system and relevant users or institutions, providing corresponding operation entrances for users to choose to agree or refuse the automated decision-making results; if the user chooses to refuse, the expert decision-making process will be entered.

[0038] The following embodiments of the present invention can be applied to various systems / applications / devices for identifying the dereliction-of-duty behaviors of safety officers. The present invention extracts a set of image frames through video, and conducts joint analysis through various methods such as target monitoring, human key point analysis, association algorithms, target tracking algorithms, and by combining the mutually exclusive relationship between personnel behaviors or the combination relationship of objects, etc., which can overcome the interference of large changes in target actions, greatly reduce false alarms, and improve the accuracy of detection results.

[0039] The present invention will be described in detail below in combination with each embodiment.

[0040] Embodiment 1

[0041] According to an embodiment of the present invention, an embodiment of a method for identifying the dereliction-of-duty behaviors of safety officers is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0042] The method embodiment provided by the first embodiment of this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the method for identifying the dereliction-of-duty behaviors of safety officers is shown. As Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ……, 102n in the figure) (the processor 102 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown therein, or have a different configuration from Figure 1 that shown.

[0043] It should be noted that the above one or more processors 102 and / or other data processing circuits are generally referred to as "data processing circuits" herein. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for identifying the behavior of a security officer's dereliction of duty in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned method for identifying the behavior of a security officer's dereliction of duty. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the above operating environment, the present application provides an Figure 2 identification method for the security officer's dereliction of duty behavior as shown. The implementation entity of this method is the identification system for the security officer's dereliction of duty behavior, which realizes the identification of the security officer's dereliction of duty behavior by combining technologies such as target identification, tracking identification, association identification, and key point identification.

[0048] Figure 2 It is a flowchart of an optional identification method for the security officer's dereliction of duty behavior according to an embodiment of the present invention. As Figure 2 shown, the method includes the following steps:

[0049] Step S101, obtain video data and preprocess the video data to obtain a set of image frames to be recognized.

[0050] It should be noted that the data source for identifying the security officer's dereliction of duty behavior in the embodiments of the present invention is the video data received from video monitoring facilities. These monitoring facilities may include visible light cameras, infrared cameras, laser-assisted lighting cameras, etc., which are deployed in areas where the behavior of security officers needs to be monitored, such as fire monitoring centers, petrochemical plant control rooms, construction site monitoring points, etc. The acquisition of video data is the basis of the entire dereliction of duty behavior real-time analysis and monitoring system, ensuring high-quality input data for subsequent image processing and behavior analysis.

[0051] In the above step S101, the acquired video data needs to go through a preprocessing step to improve the accuracy and efficiency of subsequent image recognition. The preprocessing includes, but is not limited to: frame rate adjustment. According to the processing capacity and requirements of the system, the frame rate of the video can be adjusted to ensure that it can meet the requirements of real-time monitoring without excessive consumption of computing resources. Image enhancement. The video frames are enhanced, such as adjusting brightness and contrast, denoising, sharpening, etc., to improve the image quality, especially the image quality in environments with poor lighting conditions at night. Cropping and scaling. The image is appropriately cropped to remove irrelevant background information, and at the same time, the image may need to be scaled to the size required by the algorithm to adapt to subsequent object detection and key point detection. Format conversion. Ensure that the format of the video frames is compatible with the subsequent algorithm. Usually, it needs to be converted to grayscale or a specific color space, such as RGB or HSV format. Frame decomposition. The processed video is decomposed to obtain a series of image frames, thus forming an image frame set. Frame decomposition can be carried out at a fixed frame rate, such as 30 frames per second, or the frame rate can be dynamically adjusted according to actual needs to balance real-time performance and the use of computing resources.

[0052] The purpose of the preprocessing step is to improve the detectability and recognition accuracy of the targets in the image, reduce the computational amount of subsequent processing, and provide a set of clear and high-quality image frames for the recognition of the security guard's dereliction of duty behavior.

[0053] Step S102: Perform object recognition on the image frames in the image frame set to obtain a first recognition result.

[0054] In the above step S102, through object detection technology, it is judged whether there is a security guard in the video frame, and based on this, it is judged whether the security guard is on duty and whether there is an off-duty behavior. The above first recognition result is used to indicate whether the security guard is on duty. Object recognition can be achieved through an object recognition model. The object recognition model can identify and locate multiple object categories in the image through training on a large number of labeled data sets. According to the above analysis, the system will generate a recognition result for each frame of the image, indicating whether the security guard in that frame is on duty. If on duty, the result will mark the position of the security guard and identify other dereliction of duty behaviors; if not on duty, it will further identify whether there is an off-duty behavior of the security guard, and the off-duty behavior is determined according to the length of time the security guard is not on duty.

[0055] Further, perform object recognition on the image frames in the image frame set, and the obtained first recognition result includes: performing object recognition on the image frames in the image frame set based on preset recognition features to identify whether there is a safety officer in the preset detection area; in the case that there is no safety officer in the preset detection area of the target image frame, identifying the off-duty behavior of the safety officer based on adjacent image frames, and generating an off-duty recognition result for the safety officer with off-duty behavior in the case of determining that the safety officer has off-duty behavior; or, in the case that there is a safety officer in all the image frames in the image frame set, determining that the safety officer has no off-duty behavior and generating an on-duty recognition result for the safety officer without off-duty behavior; generating the first recognition result based on the off-duty recognition result or the on-duty recognition result.

[0056] When performing object recognition, the image frame set obtained from step S101 will be processed one by one. Each image frame will be input into the object recognition algorithm, and the algorithm will output a series of object bounding boxes. Each box contains the position information and confidence score of the safety officer, indicating the probability that the object within the object bounding box is a safety officer. Analyze the object bounding boxes detected in each image frame. If a safety officer is detected in the preset detection area, it is considered that the safety officer is on duty. The preset detection area can be defined based on the historical data of the safety officer's work site and ground markings.

[0057] If a safety officer cannot be identified in the preset detection area of a certain image frame, the system will not immediately determine it as off-duty, but will confirm whether the safety officer really has an off-duty behavior based on the tracking recognition of adjacent image frames.

[0058] If a safety officer can be identified in all the image frames in the image frame set, that is, a safety officer continuously exists in the preset detection area, the system will confirm that the safety officer has no off-duty behavior. At this time, the system will generate an on-duty recognition result for the safety officer.

[0059] The system will generate the final first recognition result based on the off-duty recognition result or the on-duty recognition result, which clearly indicates the on-duty status of the safety officer during the monitoring period. The first recognition result will be used as the input for subsequent steps to further analyze whether the safety officer has other dereliction-of-duty behaviors such as sleeping on the job, playing with mobile phones, or making calls with the earpiece.

[0060] In another optional embodiment, in the case that the first recognition result indicates that the safety officer has an off-duty behavior, call the log data to obtain the emergency events reported during the off-duty period of the safety officer. In the case of determining that there is no reported emergency event for the safety officer, correct the first recognition result based on the emergency events.

[0061] Through the above embodiments, it is possible to effectively distinguish the on-duty and off-duty states of safety officers, improving the accuracy and robustness of the monitoring of slack-off behavior. The object recognition based on the preset detection area ensures targeted detection of image frames, while the analysis of consecutive image frames avoids false alarms caused by single-frame misdetection or environmental factors. At the same time, through tracking and recognition, it is possible to accurately judge the off-duty behavior of safety officers, rather than just temporary occlusion or line-of-sight blind spots. This strategy can significantly reduce the false alarm rate and improve the system stability in practical applications, ensuring that only real off-duty behaviors are marked and recorded.

[0062] Further, in the case where there is no safety officer in the preset detection area of the target image frame, the steps of identifying the off-duty behavior of the safety officer based on adjacent image frames include: when there is no safety officer in the preset detection area of the target image frame, obtaining the adjacent image frames corresponding to the target image frame, and detecting whether there is a safety officer in the preset detection areas of G adjacent image frames, where G is a positive integer; when there is no safety officer in all G adjacent image frames, determining that the safety officer has a suspected off-duty behavior, and calculating the suspected off-duty duration based on the time stamps corresponding to the image frames; comparing the suspected off-duty duration with a preset off-duty duration threshold to obtain a first comparison result; and when the first comparison result indicates that the suspected off-duty duration is greater than the off-duty duration threshold, determining that the safety officer has an off-duty behavior.

[0063] Furthermore, through the tracking and recognition of adjacent image frames, the in-depth recognition of the safety officer's off-duty behavior is carried out to improve the detection accuracy and reduce false alarms. When no safety officer is recognized in a certain image frame (target image frame), it will not be immediately determined as an off-duty behavior. Instead, a further verification process for the off-duty behavior is initiated. Automatically obtain G adjacent image frames before and after the target image frame, where G is a preset positive integer, used to form a continuous image sequence. These G image frames will be used to determine whether the safety officer has truly left the post or has temporarily not been recognized due to environmental factors (such as occlusion, lighting problems). Through continuous monitoring, the system can more accurately identify the behavior state of the safety officer and avoid false alarms caused by single-frame false positives. Perform target recognition on the G adjacent image frames to check whether there is a safety officer within the adjacent video frames. If the safety officer is continuously missing within the preset detection area in these G image frames, the system will determine that the safety officer has a suspected off-duty behavior. At this time, the system will record the start image frame when the safety officer is first not detected and the end image frame when the safety officer is detected again, and calculate the suspected off-duty duration based on the timestamps corresponding to the start image frame and the end image frame. Compare the calculated suspected off-duty duration with the preset off-duty duration threshold to generate a first comparison result. The preset off-duty duration threshold refers to the minimum duration standard considered by the system to constitute an actual off-duty behavior. If the system finds that the suspected off-duty duration exceeds this threshold, that is, the first comparison result indicates that the suspected off-duty duration is greater than the off-duty duration threshold, the system will finally determine that the safety officer has an off-duty behavior. This step ensures that only when the safety officer has been away from the post for a sufficient long time will it be determined as an off-duty behavior, thus avoiding false alarms caused by short-term absences or occlusions.

[0064] Through the monitoring of the above continuous image frames, the precise calculation of the suspected off-duty duration, and the comparison with the off-duty duration threshold, it is ensured that only when the safety officer has truly left the post and the duration reaches the threshold will it be determined as an off-duty behavior by the system. This determination method greatly reduces the false alarm rate.

[0065] Step S103, in the case where the first recognition result indicates that the safety officer is on duty in the image frame, perform tracking and recognition on the safety officer in the image frame based on the target tracking algorithm to obtain a second recognition result.

[0066] In the case where the first recognition result indicates that the safety officer is on duty in the image frame, it is necessary to further identify the slacking-off behavior of the on-duty safety officer. In the above step S103, through the target tracking algorithm, identify whether the safety officer has the slacking-off behavior of sleeping on duty to obtain a second recognition result. The target tracking algorithm tracks the same safety officer in continuous image frames and can maintain the consistency of the target even in the case of occlusion or lighting changes.

[0067] Using a target tracking algorithm to continuously track on-duty safety officers requires analyzing the continuity of the positions and behaviors of the safety officers identified in each frame of the image. The target tracking algorithm will, based on the position information of the safety officer in the previous frame, search for the target in the current frame that is most likely to be the same safety officer, and determine the tracking target by calculating the similarity of the positions between the targets and the similarity of appearance features (such as color and texture).

[0068] During the process of continuously tracking the safety officer, the system will further analyze the behavior characteristics of the safety officer to determine whether they exhibit sleeping behavior. The characteristics of sleeping behavior include the continuous immobility of the head posture, eye closure state, and body posture. A deep learning model can be used to analyze the target area to identify the head and body postures of the safety officer, as well as the state of the eyes, and thus monitor whether the safety officer is in a sleeping state.

[0069] Furthermore, in the case where the first recognition result indicates that the safety officer is on duty in the image frame, the steps of performing tracking and recognition on the safety officer in the image frame based on the target tracking algorithm to obtain the second recognition result include: in the case where the first recognition result indicates that the safety officer is on duty in the image frame, performing pose analysis on the safety officer in the target image frame to determine whether the safety officer is in a sleeping state, where the pose analysis at least includes: head tilt degree analysis, eye closure degree analysis, and body pose state analysis; in the case where the safety officer is in a sleeping state, performing tracking and recognition on the safety officer in the adjacent image frames corresponding to the target image frame, and in the case where the safety officers in K adjacent image frames are all in a sleeping state, calculating the sleeping duration of the safety officer, where K is a positive integer; comparing the sleeping duration with a preset sleeping duration threshold to obtain a second comparison result, and in the case where the second comparison result indicates that the sleeping duration is greater than the sleeping duration threshold, determining that the safety officer has the behavior of sleeping on duty, generating a sleeping behavior recognition result for the safety officer with the behavior of sleeping on duty, and obtaining the second recognition result based on the sleeping behavior recognition result.

[0070] Specifically, in the case where it is determined that the safety officer in the image frame is on duty, perform pose analysis on the safety officer in the target image frame to identify whether they are in a sleeping state. The pose analysis is performed by detecting human key points, including the head, eyes, and other parts of the body. Specifically, it includes: head tilt degree analysis, by identifying the position and pose of the safety officer's head, analyzing whether the head maintains an angle lower than the normal level for a long time, which usually indicates that they may be sleeping; eye closure degree analysis, using the key point information of the eye area, such as the position of the eyelids, to determine whether the eyes are closed or the degree of closure; body pose state analysis, combining the analysis of the head and eyes, and the remaining body key points detected, such as the positions of the arms and legs, to comprehensively judge whether the overall pose of the safety officer conforms to a lying or other typical sleeping pose.

[0071] After determining that the safety officer in the target image frame is in a sleeping state, start the target tracking algorithm to continuously track and identify the safety officer in the target image frame and its adjacent K image frames. K is a positive integer, and the value is selected based on the time length required to accurately determine whether the safety officer continuously maintains a sleeping state. The target tracking algorithm can ensure that the system tracks the same safety officer between different image frames and maintain the coherence of recognition even in complex backgrounds or occlusion situations.

[0072] Through continuous K-frame tracking verification, the system can confirm whether the safety officer continuously maintains a sleeping state in multiple image frames, avoiding misidentifications caused by short-term posture changes or false alarms. If the safety officer is identified as being in a sleeping state in all K frames, then calculate the specific duration of the safety officer sleeping on duty to obtain the sleeping duration. If the sleeping duration exceeds the pre-set duration threshold, it is determined that the safety officer has the behavior of sleeping on duty, generate a recognition result of the sleeping behavior, and thus obtain the second recognition result. Conversely, if the sleeping duration does not reach the pre-set sleeping duration threshold or there is no sleeping safety officer in the image frame, generate a recognition result of no sleeping behavior, and thus obtain the second recognition result.

[0073] Step S104, perform target detection on the image frame set to obtain a target detection result.

[0074] Furthermore, through steps S101 - S103, the off-duty behavior and sleeping-on-duty behavior of the safety officer can be identified. In addition, the slacking-off behavior also includes the behavior of making phone calls and playing with mobile phones. In the above step S104, by performing target detection on the image frame set, it is identified whether there is a handheld terminal in the image frame to obtain a detection result, and the detection result is the basis for identifying the behavior of making phone calls and playing with mobile phones.

[0075] Step S105, in the case where the target detection result indicates that there is a handheld terminal in the image frame, perform associated recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result.

[0076] In the above step S105, when a handheld terminal is detected in a certain image frame, an associated recognition process will be started. The purpose of the associated recognition is to determine whether the handheld terminal is associated with the on-duty safety officer, that is, to judge whether the safety officer is holding or using the terminal. The associated recognition can use the Hungarian algorithm, an algorithm for finding the best match in multiple choices, to solve the problem of the association between people and objects and ensure that the handheld terminal is correctly associated with a specific safety officer.

[0077] Further, in the case where the object detection result indicates the presence of a handheld terminal in the image frame, the steps of performing associated recognition and key-point recognition on the safety officer and the handheld terminal to obtain a third recognition result include: in the case where there is a handheld terminal in the image frame, using the Hungarian algorithm to perform associated recognition on the safety officer and the handheld terminal in the target image frame with the handheld terminal to obtain a first associated recognition result, and the associated recognition includes recognizing the relative position and relative distance between the safety officer and the handheld terminal; in the case where the first associated recognition result indicates an association between the safety officer and the handheld terminal, using the Hungarian algorithm to perform associated recognition on the handheld terminal and the hand key points of the safety officer to obtain a second associated recognition result; in the case where the second associated recognition result indicates an association between the handheld terminal and the hand key points of the safety officer, recognizing the relative relationship between the handheld terminal and the target key points of the safety officer to obtain a third recognition result, where the target key points include at least one of the following: ear key points, eye key points.

[0078] Specifically, when identifying the dereliction of duty behavior of the safety officer using the handheld terminal, it includes two parts: associated recognition and key-point recognition. In associated recognition, the handheld terminal is first associated with the safety officer, and in the case of successful association, the handheld terminal is then associated with the hand key points of the safety officer. The associated recognition includes the following steps: target extraction, extracting the bounding box information of the handheld terminal and the position information of the object from the object detection result; calculating the distance, measuring the relative distance between each object and the handheld terminal to determine the degree of proximity; matching analysis, using the Hungarian algorithm to find the best match between the handheld terminal and the object, that is, finding the object with the closest distance. If the closest object is the safety officer, the association between the handheld terminal and the safety officer is successful. Finally, based on the matching analysis, a first associated recognition result indicating whether the handheld terminal and the safety officer are associated is generated.

[0079] Further, in the case where the handheld terminal is associated with the safety officer, the handheld terminal is further associated with the hand key points of the safety officer to determine whether the safety officer is holding the handheld terminal. Specifically, it includes: hand key-point extraction, extracting the coordinates of each key point of the safety officer's hand from the human pose estimation result; calculating the relative position, determining the relative relationship between the position of the handheld terminal in the image and the hand key points; matching analysis, using the Hungarian algorithm to find the best match between the handheld terminal and the hand key points to further confirm whether the safety officer is using the handheld terminal. Finally, based on the matching analysis, a second associated recognition result indicating whether the handheld terminal and the hand key points are associated is generated.

[0080] When the second associated recognition result indicates an association between the handheld terminal and the hand key points of the hand, the system will perform a deeper analysis to determine whether the safety officer is performing behaviors such as playing with the mobile phone or holding the phone to the ear. This step involves the recognition of the relative position relationship between the target key points (including: ear key points, eye key points) and the handheld terminal. Based on the relative position analysis, a recognition result is generated that clearly indicates whether the handheld terminal has a specific relative relationship with the ear and eye target key points of the safety officer, which is used to determine whether the safety officer is holding the phone to the ear or playing with the mobile phone, and a third recognition result is generated.

[0081] Further, the steps for recognizing the relative relationship between the handheld terminal and the target key points of the safety officer include: obtaining the largest acute angle in the triangular relationship formed by the eye key points of the safety officer and the handheld terminal; comparing the angle value of the largest acute angle with a preset angle threshold to obtain a third comparison result; when the third comparison result indicates that the angle value of the largest acute angle is less than the preset angle threshold, determining that the safety officer has a first suspected terminal usage behavior, and obtaining the target image frame in which the safety officer has the first suspected terminal usage behavior; based on the adjacent image frames corresponding to the target image frame, performing tracking recognition on the safety officer with the first suspected terminal usage behavior, and when the safety officers in M adjacent image frames all have the first suspected terminal usage behavior, calculating the first terminal usage duration of the safety officer, where M is a positive integer; when the first terminal usage duration is greater than the preset terminal usage duration threshold, determining that the safety officer has the behavior of using the handheld terminal for entertainment.

[0082] Specifically, the recognition of the relative relationship of the key points includes the triangular relationship formed by the eye key points and the handheld terminal associated with the hand key points of the safety officer, so as to deeply determine whether the safety officer is using the handheld terminal for entertainment. Based on the line connecting the two eye key points and the lines connecting the two eye key points to the handheld terminal respectively, a triangular relationship is obtained, and then the largest acute angle is selected, and the angle value of the largest acute angle is compared with the preset angle value. If the angle value of the largest acute angle is less than the preset angle value, it is determined that the safety officer has a first suspected terminal usage behavior, that is, the behavior of playing with the mobile phone.

[0083] Furthermore, when there is a suspected terminal usage behavior, the system will continuously track and identify the safety officer in the target image frame and its adjacent M image frames based on the target tracking algorithm to confirm the persistence of the suspected behavior. M is a positive integer determined according to specific application scenarios and the needs of behavior recognition. The continuous tracking and recognition of M frames can not only verify the authenticity of the suspected terminal usage behavior but also calculate the specific duration of the safety officer performing this behavior. The system calculates the first terminal usage duration based on the tracking and recognition results, and this duration is calculated based on the timestamps of the image frames. Through the timestamps, the system can accurately record the duration from the start to the end of the suspected behavior, providing quantitative data support for subsequent behavior confirmation.

[0084] Finally, compare the calculated first terminal usage duration with a preset terminal usage duration threshold. The preset terminal usage duration threshold is set based on the analysis of the duration of using the terminal for entertainment during normal working hours and is used to distinguish between the behavior of briefly viewing the terminal normally and using the terminal for long-term entertainment. If the first terminal usage duration is greater than the preset duration threshold, the system will determine that the safety officer has the behavior of using the handheld terminal for entertainment, that is, the first terminal usage behavior.

[0085] Furthermore, the step of identifying the relative relationship between the handheld terminal and the target key points of the safety officer further includes: obtaining the distance value between the ear key point of the safety officer and the handheld terminal; comparing the distance value with a preset distance threshold to obtain a fourth comparison result; in the case where the fourth comparison result indicates that the distance value is less than the preset distance threshold, determining that the safety officer has a second suspected terminal usage behavior and obtaining the target image frame when the safety officer has the second suspected terminal usage behavior; based on the adjacent image frames corresponding to the target image frame, tracking and identifying the safety officer with the second suspected terminal usage behavior. In the case where the safety officer in all N adjacent image frames has the second suspected terminal usage behavior, calculate the second terminal usage duration of the safety officer, where N is a positive integer; in the case where the second terminal usage duration is greater than the preset terminal usage duration threshold, determine that the safety officer has the behavior of using the handheld terminal to make a call.

[0086] In some embodiments, the recognition of the relative relationship of key points further includes the recognition of the distance relationship between the key points of the ear and the handheld terminal associated with the key points of the safety officer's hand. Specifically, first, calculate the distance value between the key points of the ear and the handheld terminal. This distance calculation can be obtained based on the Euclidean distance formula between two points, aiming to quantify the proximity between the handheld terminal and the ear, so as to determine whether the terminal is being used for making a call. The calculated distance value will be compared with the preset distance threshold in the system to obtain the fourth comparison result. The preset distance threshold is set based on the statistical analysis of the average distance between the handheld terminal and the ear during a call. If the fourth comparison result indicates that the distance value is less than the preset distance threshold, it means that the proximity between the handheld terminal and the key points of the ear conforms to the typical behavioral characteristics of making a call, and the system will preliminarily determine that the safety officer may have a second suspected terminal usage behavior, that is, the behavior of making a call.

[0087] Further, after preliminarily determining that the safety officer has a second suspected terminal usage behavior, the system will perform tracking and recognition on the safety officer in the target image frame and its continuous N adjacent image frames based on the target tracking algorithm. N is a positive integer, and the value selection depends on the number of image frames required to confirm the continuity of the behavior. The tracking and recognition of N consecutive frames can verify the authenticity of the suspected behavior. According to the tracking and recognition results, calculate the second terminal usage duration, that is, the duration of the safety officer making a call. This duration calculation is crucial for accurately identifying the dereliction of duty behavior of making a long call. Then, compare the calculated second terminal usage duration with the preset terminal usage duration threshold. The preset terminal usage duration threshold is set based on the duration standard of making a call during normal working hours, and is used to distinguish between short necessary calls and long non-necessary call behaviors. If the second terminal usage duration is greater than the preset duration threshold, the system will determine that the safety officer has the behavior of using the handheld terminal to make a call, that is, the second terminal usage behavior. This confirmation based on the quantitative index of duration ensures the rigor and accuracy of the recognition.

[0088] Through the above steps, by analyzing the relative positional relationship between the handheld terminal and the key points of the safety officer's ear and eye, and combining the tracking and recognition of continuous image frames, the in-depth recognition of the safety officer's handheld terminal usage behavior is achieved. It not only improves the accuracy of the system in specific behavior recognition, but also can effectively distinguish the boundary between the safety officer's normal call and dereliction of duty behavior.

[0089] Step S106, generate a dereliction of duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result.

[0090] Based on the above step S106, multiple recognition algorithms such as object detection, tracking recognition, association recognition, and key point recognition are used to detect and analyze different idling behaviors of the safety officer, so as to obtain a first recognition result indicating whether the safety officer is absent from his post, a second recognition result indicating whether the safety officer is sleeping on the job, and a third recognition result indicating whether the safety officer is playing with a mobile phone or making a call on the job. The multiple recognition results are integrated to generate an idling behavior recognition result.

[0091] Further, the step of generating an idling behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result further includes: when the third recognition result indicates that the safety officer has a handheld terminal usage behavior while on the job, the second recognition result is corrected based on the third recognition result to obtain a corrected second recognition result; an idling behavior recognition result is generated based on the first recognition result, the corrected second recognition result, and the third recognition result.

[0092] In some embodiments, when generating an idling behavior recognition result, the recognition result is corrected based on the mutual exclusivity between behaviors. Considering the mutual exclusivity that a person making a call or playing with a mobile phone will not sleep at the same time, when the third recognition result indicates that the safety officer has a mobile phone usage or earphone call behavior while on the job, the system will correct the second recognition result based on the principle of mutual exclusivity between behaviors. If the second recognition result has misidentified the safety officer as being in a sleeping state, and at this time the third recognition result indicates that the safety officer is using a handheld terminal, the system will update the second recognition result and correct the state of the safety officer to avoid misreporting a safety officer who is playing with a mobile phone or making a call as being in a sleeping state. In addition, the system can also dynamically adjust the time threshold for sleeping on the job according to the duration of the handheld terminal usage behavior to ensure that a safety officer who is playing with a mobile phone or making a call will not be misjudged as being in a sleeping state even if he closes his eyes to rest. Finally, the corrected results are integrated to generate an idling behavior recognition result of the safety officer.

[0093] Through the above steps, video data is obtained and preprocessed to obtain a set of image frames to be recognized; target recognition is performed on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether a safety officer is on duty; in the case where the first recognition result indicates that the safety officer is on duty in the image frame, based on the target tracking algorithm, the safety officer in the image frame is tracked and recognized to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has a behavior of sleeping on duty; target detection is performed on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; in the case where the target detection result indicates that there is a handheld terminal in the image frame, correlation recognition and key point recognition are performed on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a behavior of using the handheld terminal; a dereliction-of-duty behavior recognition result is generated based on the first recognition result, the second recognition result, and the third recognition result to identify the dereliction-of-duty behavior of the safety officer.

[0094] In this embodiment, through multiple recognition technologies such as target detection, tracking recognition, correlation recognition, and key point recognition, target detection and correlation recognition are performed on multiple image frames in the set of image frames, reducing false alarms caused by single recognition methods and single recognition content, achieving fast and accurate recognition of the dereliction-of-duty behavior of safety officers, improving the accuracy of the dereliction-of-duty behavior recognition result, and thus solving the technical problem in the related art that the method for recognizing the dereliction-of-duty behavior of safety officers based on target detection relies on single-frame images and has a low accuracy of recognition results.

[0095] The following is a detailed description in combination with another optional specific implementation manner.

[0096] Figure 3 is an optional architecture diagram of a system for recognizing the dereliction-of-duty behavior of safety officers according to an embodiment of the present invention, as Figure 3 shown, the system for recognizing the dereliction-of-duty behavior of safety officers is composed of video monitoring facilities, a network, a server, and a client. Specifically, the video monitoring facilities capture videos and transmit them to the algorithm server in real time through the network for personnel dereliction-of-duty behavior recognition. According to the recognition result, warning information is sent to the business processor for warning disposal, and finally the warning information is sent to the customer processing interface and the feedback of the warning disposal by the customer processing interface is received.

[0097] Specifically, the main dereliction-of-duty behaviors of safety officers in the accident area include making phone calls, playing with mobile phones, sleeping on duty, leaving the post, etc.; the video monitoring facilities are mainly visible light dome cameras, visible light bullet cameras, and brackets or bases for installing cameras.

[0098] Figure 4It is a schematic diagram of an optional identification process for the slacking behavior of safety officers according to an embodiment of the present invention, corresponding to the part of identifying the slacking behavior of safety officers by the algorithm server, as follows Figure 4 As shown, the identification process of the slacking behavior of safety officers includes:

[0099] Step 1: Read the video. When receiving the real-time analysis instruction for the slacking behavior of safety officers and the video data, start the behavior analysis and judgment method, and decompose the original video into individual image frames.

[0100] Step 2: Object detection. Use the object detection algorithm to identify the safety officer and the handheld terminal from the image frames, and at the same time identify the key points of the hand, ear, and eye.

[0101] Step 3: Determine whether there is a safety officer. If not, execute Step 4; if so, execute Step 5.

[0102] Step 4: Perform target tracking and identification. Track and identify adjacent image frames without a safety officer, and determine whether there is a safety officer in multiple adjacent image frames. If none exist, calculate the off-duty time of the safety officer and determine the off-duty behavior of the safety officer.

[0103] Step 5: Determine whether there is a sleeping behavior through pose analysis. If so, execute Step 6; if not, execute Step 7.

[0104] Step 6: Perform target tracking on the safety officer with a sleeping behavior, and identify whether there is a sleeping behavior in multiple image frames. If so, calculate the sleeping duration, and determine whether the safety officer actually has a sleeping-on-duty behavior based on the sleeping duration.

[0105] When a safety officer has a sleeping behavior, use the target tracking algorithm to track the sleeping person, continuously track the target in subsequent video frames, and at the same time calculate the sleeping duration of the sleeping person based on the tracking results. Determine whether the sleeping duration of the sleeping person has reached the sleeping duration for sleeping on duty. If it has reached, issue an alarm; otherwise, process the next frame of the video.

[0106] Step 7: Determine whether there is a handheld terminal in the image frame. If so, execute Step 8; if not, directly skip the subsequent identification and execute Step 11.

[0107] Step 8: Perform association identification.

[0108] Step 9: Determine whether it is associated with the safety officer. If so, execute Step 10; if not, execute Step 11.

[0109] If there is a mobile phone in the image frame, the Hungarian algorithm is used to associate all the detected mobile phones and safety officers in the current frame, and it is judged whether the mobile phone can be associated with the safety officer. If not, it is considered an unused mobile phone, and the next frame of the video is processed. If it is associated with the safety officer, the next judgment is made.

[0110] Step 10: Conduct key point recognition, mainly including hand key points and ear key points, and determine whether the safety officer is holding the phone to the ear. If so, analyze the ear and hand key points to determine whether there is a phone call behavior. On the other hand, determine whether the safety officer is holding a mobile phone. If so, conduct eye and hand key point analysis and confirm whether the safety officer has the behavior of playing with the mobile phone.

[0111] For the person associated with the mobile phone, use the Hungarian algorithm to associate the handheld terminal with the hand holding the mobile phone. If the hand holding the mobile phone is associated, conduct human key point detection, and based on the positional relationship between the eye key points and the handheld terminal, determine whether the safety officer is using the mobile phone. When it is determined that the safety officer is using the mobile phone, use the target tracking algorithm to track the safety officer, and determine whether it is associated with the safety officer who has used the mobile phone at the previous moment. If it is associated, determine whether the usage duration of the mobile phone by this safety officer has reached the warning threshold duration for playing with the mobile phone. If it has reached, give an alarm; otherwise, process the next frame of the video.

[0112] For the safety officer associated with the mobile phone, use the Hungarian algorithm to associate the handheld terminal with the ear key points of the safety officer. If the ear key points are associated, use the target tracking algorithm to track the safety officer in multiple image frames, and determine whether it is associated with the safety officer who has made a phone call at the previous moment. If it is associated, determine whether the call duration of this safety officer has reached the warning threshold duration for making a phone call. If it has reached, give an alarm; otherwise, process the next frame of the video.

[0113] Step 11: Determine the slack-off behavior of the safety officer based on the recognition of the safety officer's off-duty behavior, the recognition of the safety officer's sleeping-on-duty behavior, and the recognition of the safety officer's phone call behavior and mobile phone playing behavior, and generate the recognition result of the slack-off behavior.

[0114] Figure 5 It is a schematic diagram of an optional method for recognizing the slack-off behavior of a safety officer based on eye key points according to an embodiment of the present invention. As Figure 5 shown, when recognizing the slack-off behavior of a safety officer based on eye key points, according to the straight line formed by two eye key points and the maximum acute angle formed by the straight lines from the two eye key points to the center point of the handheld terminal respectively, it is judged whether the acute angle is greater than a preset threshold. If it is greater than the preset threshold, it is considered that the mobile phone is held but not played with; otherwise, it is determined that the phone is being played with. Specifically, as Figure 5As shown, a connection line is formed between the eye key points 123 and the eye key points 654. At the same time, the eye key points 123 and the eye key points 654 are respectively connected to the mobile phone, thus forming a triangular relationship, and identifying the largest acute angle in the relationship. Figure 5 It is composed of the largest acute angle 1, the largest acute angle 2, and the largest acute angle 3. Whether the safety officer is playing with the mobile phone is determined by the value of the largest acute angle.

[0115] Figure 6 It is a schematic diagram of an optional method for identifying the dereliction of duty behavior of a safety officer based on ear key points according to an embodiment of the present invention. As Figure 6 shown, when identifying the behavior of the safety officer making a call based on the ear key points, according to the distance relationship between two ear key points ( Figure 6 schematically shown by the ear key points 7 and the ear key points 8 in the figure) and the hand key points holding the mobile phone ( Figure 6 schematically shown by the hand key points 15, 16, 17, 18, 19, 20, 21, and 22 in the figure), it is judged whether the distance value is less than the specified distance threshold. If it is less than the specified distance threshold, it is considered that the safety officer has the behavior of holding the phone to the ear and making a call.

[0116] In the embodiment of the present invention, by extracting an image frame set from a video and jointly analyzing the image frame set through various methods such as target monitoring, human key point analysis, association algorithms, target tracking algorithms, and combining the mutual exclusion between personnel behaviors or the combination relationship of objects, the interference of large changes in target actions can be overcome, the false alarm rate can be greatly reduced, and the accuracy of the detection result can be improved.

[0117] The following is a detailed description in combination with another embodiment.

[0118] Embodiment 2

[0119] The identification device for the dereliction of duty behavior of a safety officer provided in this embodiment includes multiple implementation units. Each implementation unit corresponds to each implementation step in the first embodiment above. The specific implementation manner and beneficial effects can be referred to the foregoing method embodiment and will not be elaborated here.

[0120] Figure 7 It is a schematic diagram of an optional identification device for the dereliction of duty behavior of a safety officer according to an embodiment of the present invention. As Figure 7 shown, the identification device for the dereliction of duty behavior of a safety officer may include: an acquisition unit 71, a first identification unit 72, a second identification unit 73, a hair detection unit 74, a third identification unit 75, and a generation unit 76, where

[0121] The acquisition unit 71 is used to acquire video data and preprocess the video data to obtain an image frame set to be identified;

[0122] The first recognition unit 72 is configured to perform target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether a safety officer is on duty;

[0123] The second recognition unit 73 is configured to, when the first recognition result indicates that the safety officer is on duty in the image frame, perform tracking recognition on the safety officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has a behavior of sleeping on duty;

[0124] The detection unit 74 is configured to perform target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether a handheld terminal exists in the image frame;

[0125] The third recognition unit 75 is configured to, when the target detection result indicates that a handheld terminal exists in the image frame, perform associated recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a behavior of using the handheld terminal;

[0126] The generation unit 76 is configured to generate an off-duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result.

[0127] The above-mentioned device for recognizing the off-duty behavior of a safety officer obtains video data through the acquisition unit 71, preprocesses the video data to obtain a set of image frames to be recognized; performs target recognition on the image frames in the set of image frames through the first recognition unit 72 to obtain a first recognition result, where the first recognition result is used to indicate whether a safety officer is on duty; when the first recognition result indicates that the safety officer is on duty in the image frame, performs tracking recognition on the safety officer in the image frame based on a target tracking algorithm through the second recognition unit 73 to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has a behavior of sleeping on duty; performs target detection on the set of image frames through the detection unit 74 to obtain a target detection result, where the target detection result is used to indicate whether a handheld terminal exists in the image frame; when the target detection result indicates that a handheld terminal exists in the image frame, performs associated recognition and key point recognition on the safety officer and the handheld terminal through the third recognition unit 75 to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a behavior of using the handheld terminal; generates an off-duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result through the generation unit 76.

[0128] In this embodiment, through multiple recognition technologies such as target detection, tracking recognition, association recognition, and key point recognition, target detection and association recognition are performed on multiple image frames in the image frame set, reducing false alarms caused by single recognition methods and single recognition content, achieving fast and accurate recognition of the absent-on-duty behavior of safety officers, improving the accuracy of the absent-on-duty behavior recognition results, and thus solving the technical problem in the related art that the method for recognizing the absent-on-duty behavior of safety officers based on target detection relies on single-frame images and has a low accuracy of recognition results.

[0129] Further, the first recognition unit 72 includes: a first recognition module for performing target recognition on the image frames in the image frame set based on preset recognition features to recognize whether there is a safety officer in the preset detection area; a second recognition module for, when there is no safety officer in the preset detection area of the target image frame, recognizing the absent-on-duty behavior of the safety officer based on adjacent image frames, and generating an absent-on-duty recognition result for the safety officer with an absent-on-duty behavior when it is determined that the safety officer has an absent-on-duty behavior; a third recognition module for, when there is a safety officer in all the image frames in the image frame set, determining that the safety officer has no absent-on-duty behavior and generating a on-duty recognition result for the safety officer without an absent-on-duty behavior; a first generation module for generating a first recognition result based on the absent-on-duty recognition result or the on-duty recognition result.

[0130] Further, the second recognition module 73 includes: a first acquisition sub-module for, when there is no safety officer in the preset detection area of the target image frame, acquiring the adjacent image frame corresponding to the target image frame and detecting whether there is a safety officer in the preset detection area of G adjacent image frames, where G is a positive integer; a first determination sub-module for, when there is no safety officer in all G adjacent image frames, determining that the safety officer has a suspected absent-on-duty behavior and calculating the suspected absent-on-duty duration based on the time stamp corresponding to the image frame; a first comparison sub-module for comparing the suspected absent-on-duty duration with a preset absent-on-duty duration threshold to obtain a first comparison result; a second determination sub-module for, when the first comparison result indicates that the suspected absent-on-duty duration is greater than the absent-on-duty duration threshold, determining that the safety officer has an absent-on-duty behavior.

[0131] Further, the second recognition unit 73 includes: a first analysis module, configured to perform pose analysis on the safety officer in the target image frame to determine whether the safety officer is in a sleeping state when the first recognition result indicates that the safety officer is on duty in the image frame, where the pose analysis at least includes: head tilt degree analysis, eye closure degree analysis, and body posture state analysis; a fourth recognition module, configured to track and recognize the safety officer in the adjacent image frames corresponding to the target image frame when the safety officer is in a sleeping state, and calculate the sleeping duration of the safety officer when the safety officers in K adjacent image frames are all in a sleeping state, where K is a positive integer; a second comparison module, configured to compare the sleeping duration with a preset sleeping duration threshold to obtain a second comparison result, and determine that the safety officer has a behavior of sleeping on duty, generate a sleeping behavior recognition result for the safety officer with the behavior of sleeping on duty, and obtain a second recognition result based on the sleeping behavior recognition result.

[0132] Further, the third recognition unit 75 includes: a fifth recognition module, configured to perform association recognition on the safety officer and the handheld terminal in the target image frame with the handheld terminal using the Hungarian algorithm when there is a handheld terminal in the image frame, and obtain a first association recognition result, where the association recognition includes recognizing the relative position and relative distance between the safety officer and the handheld terminal; a sixth recognition module, configured to perform association recognition on the handheld terminal and the hand key points of the safety officer using the Hungarian algorithm when the first association recognition result indicates an association between the safety officer and the handheld terminal, and obtain a second association recognition result; a seventh recognition module, configured to recognize the relative relationship between the handheld terminal and the target key points of the safety officer when the second association recognition result indicates an association between the handheld terminal and the hand key points of the safety officer, and obtain a third recognition result, where the target key points include at least one of the following: ear key points, eye key points.

[0133] Further, the seventh recognition module includes: a second acquisition sub-module, configured to acquire the largest acute angle in the triangular relationship formed by the eye key points of the safety officer and the handheld terminal; a second comparison sub-module, configured to compare the angle value of the largest acute angle with a preset angle threshold to obtain a third comparison result; a third determination sub-module, configured to determine that the safety officer has a first suspected terminal usage behavior and acquire the target image frame in which the safety officer has the first suspected terminal usage behavior when the third comparison result indicates that the angle value of the largest acute angle is less than the preset angle threshold; a first calculation sub-module, configured to perform tracking and recognition on the safety officer with the first suspected terminal usage behavior based on the adjacent image frames corresponding to the target image frame, and calculate the first terminal usage duration of the safety officer when the safety officers in all of the M adjacent image frames have the first suspected terminal usage behavior, where M is a positive integer; a fourth determination sub-module, configured to determine that the safety officer has the behavior of using the handheld terminal for entertainment when the first terminal usage duration is greater than a preset terminal usage duration threshold.

[0134] Further, the seventh recognition module further includes: a third acquisition sub-module, configured to acquire the distance value between the ear key points of the safety officer and the handheld terminal; a third comparison sub-module, configured to compare the distance value with a preset distance threshold to obtain a fourth comparison result; a fifth determination sub-module, configured to determine that the safety officer has a second suspected terminal usage behavior and acquire the target image frame in which the safety officer has the second suspected terminal usage behavior when the fourth comparison result indicates that the distance value is less than the preset distance threshold; a second calculation sub-module, configured to perform tracking and recognition on the safety officer with the second suspected terminal usage behavior based on the adjacent image frames corresponding to the target image frame, and calculate the second terminal usage duration of the safety officer when the safety officers in all of the N adjacent image frames have the second suspected terminal usage behavior, where N is a positive integer; a sixth determination sub-module, configured to determine that the safety officer has the behavior of using the handheld terminal to make a call when the second terminal usage duration is greater than a preset terminal usage duration threshold.

[0135] Further, the generation unit 76 includes: a first correction module, configured to correct the second recognition result based on the third recognition result to obtain a corrected second recognition result when the third recognition result indicates that the safety officer has a handheld terminal usage behavior while on duty; a second generation module, configured to generate a dereliction-of-duty behavior recognition result based on the first recognition result, the corrected second recognition result, and the third recognition result.

[0136] It should be noted here that the above-mentioned acquisition unit 71, first recognition unit 72, second recognition unit 73, hair detection unit 74, third recognition unit 75, and generation unit 76 correspond to steps S201 to S206 in the first embodiment. The examples and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the content disclosed in the first embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory (for example, memory 104) and processed by one or more processors (for example, processors 102a, 102b,..., 102n). The above modules or units can also be part of a device and can run in the computer terminal 10 provided in the first embodiment.

[0137] The present invention will be described below in conjunction with another alternative embodiment.

[0138] Embodiment III

[0139] The embodiment of the present invention can also provide an electronic device. Figure 8 It is a hardware structure block diagram of an electronic device (or mobile device) that can optionally execute the method for identifying the behavior of a security officer's dereliction of duty according to the embodiment of the present invention, as Figure 8 shown. The electronic device may include: one or more ( Figure 8 only one is shown in the figure) processors 802, a memory 804, a storage controller, and a peripheral interface. Among them, the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0140] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned methods. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely provided with respect to the processor, and these remote memories may be connected to the terminal through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain video data, and preprocess the video data to obtain a set of image frames to be recognized; perform target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the safety officer is on duty; in the case where the first recognition result indicates that the safety officer is on duty in the image frame, perform tracking recognition on the safety officer in the image frame based on the target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has the behavior of sleeping on duty; perform target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; in the case where the target detection result indicates that there is a handheld terminal in the image frame, perform association recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has the behavior of using the handheld terminal; generate a recognition result of dereliction of duty behavior based on the first recognition result, the second recognition result, and the third recognition result.

[0142] By adopting the embodiment of the present invention, a method for recognizing the dereliction of duty behavior of a safety officer is provided. Through multiple recognition technologies such as target detection, tracking recognition, association recognition, and key point recognition, target detection and association recognition are performed on multiple image frames in the set of image frames, reducing false alarms caused by single recognition methods and single recognition contents, achieving rapid and accurate recognition of the dereliction of duty behavior of the safety officer, improving the accuracy of the recognition result of the dereliction of duty behavior, and further solving the technical problem in the related art that the method for recognizing the dereliction of duty behavior of the safety officer based on target detection depends on a single-frame image and has a low accuracy of the recognition result.

[0143] Those of ordinary skill in the art can understand that Figure 8 The structure shown is only schematic, and the electronic device can also be a terminal device such as a smart phone, a tablet computer, a personal digital assistant, and a Mobile Internet Device (MID), a PAD, etc. Figure 8 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components (such as a network interface, a display device, etc.) than those shown Figure 8 in the figure, or have a different configuration from that shown Figure 8 in the figure.

[0144] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing the hardware related to the terminal device. This program can be stored in a computer-readable storage medium, and the storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0145] The present invention will be described below in conjunction with another alternative embodiment.

[0146] Embodiment 4

[0147] The embodiment of the present invention also provides a computer-readable storage medium. Optionally, in the embodiment of the present invention, the above computer-readable storage medium can be used to store the program code executed by the method for identifying the behavior of a security officer being absent from duty provided in the first embodiment above.

[0148] Optionally, in the embodiment of the present invention, the above storage medium can be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0149] The embodiment of the present invention also provides a computer program product. When executed on a data processing device, it is adapted to execute a program for the steps of the method for identifying the behavior of a security officer being absent from duty: obtaining video data, and preprocessing the video data to obtain a set of image frames to be identified; performing target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the security officer is on duty; in the case where the first recognition result indicates that the security officer is on duty in the image frame, performing tracking recognition on the security officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the security officer has the behavior of sleeping on duty; performing target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; in the case where the target detection result indicates that there is a handheld terminal in the image frame, performing association recognition and key point recognition on the security officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the security officer has the behavior of using a handheld terminal; generating an identification result of the absent-duty behavior based on the first recognition result, the second recognition result, and the third recognition result.

[0150] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0151] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings, direct couplings, or communication connections shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0153] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0155] If the above-mentioned integrated units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0156] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for identifying the behavior of a safety officer neglecting his or her duties, characterized in that, Including: Obtain video data, and preprocess the video data to obtain a set of image frames to be recognized; Perform target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the safety officer is on duty; When the first recognition result indicates that the safety officer is on duty in the image frame, perform tracking recognition on the safety officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has a behavior of sleeping on duty; Perform target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; When the target detection result indicates that there is a handheld terminal in the image frame, perform association recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a behavior of using a handheld terminal; Generate a recognition result of dereliction of duty based on the first recognition result, the second recognition result, and the third recognition result.

2. The method according to claim 1, characterized in that, Performing target recognition on the image frames in the set of image frames to obtain a first recognition result includes: Perform target recognition on the image frames in the set of image frames based on preset recognition features to identify whether there is a safety officer in a preset detection area; When there is no safety officer in the preset detection area of the target image frame, identify the off-duty behavior of the safety officer based on adjacent image frames. When it is determined that the safety officer has an off-duty behavior, generate an off-duty recognition result for the safety officer with the off-duty behavior; or, When there is a safety officer in all the image frames in the set of image frames, determine that the safety officer has no off-duty behavior, and generate an on-duty recognition result for the safety officer without off-duty behavior; Generate the first recognition result based on the off-duty recognition result or the on-duty recognition result.

3. The method according to claim 2, characterized in that, The step of identifying the off-duty behavior of the safety officer based on adjacent image frames when there is no safety officer in the preset detection area of the target image frame includes: When there is no safety officer in the preset detection area of the target image frame, obtain the adjacent image frames corresponding to the target image frame, and detect whether there is a safety officer in the preset detection area of G adjacent image frames, where G is a positive integer; When there is no safety officer in all G adjacent image frames, determine that the safety officer has a suspected off-duty behavior, and calculate the suspected off-duty duration based on the time stamp corresponding to the image frame; Compare the suspected off-duty duration with a preset off-duty duration threshold to obtain a first comparison result; When the first comparison result indicates that the suspected off-duty duration is greater than the off-duty duration threshold, determine that the safety officer has an off-duty behavior.

4. The method according to claim 1, characterized in that When the first recognition result indicates that the safety officer is on duty in the image frame, the step of performing tracking recognition on the safety officer in the image frame based on a target tracking algorithm to obtain a second recognition result includes: When the first recognition result indicates that the safety officer is on duty in the image frame, perform pose analysis on the safety officer in the target image frame to determine whether the safety officer is in a sleeping state. Among them, the pose analysis at least includes: head tilt degree analysis, eye closure degree analysis, and body posture state analysis; When the safety officer is in a sleeping state, perform tracking and recognition on the safety officer in the adjacent image frames corresponding to the target image frame. When the safety officers in K adjacent image frames are all in a sleeping state, calculate the sleeping duration of the safety officer, where K is a positive integer; Compare the sleeping duration with a preset sleeping duration threshold to obtain a second comparison result. When the second comparison result indicates that the sleeping duration is greater than the sleeping duration threshold, determine that the safety officer has a behavior of sleeping on duty, generate a sleeping behavior recognition result for the safety officer with the behavior of sleeping on duty, and obtain a second recognition result based on the sleeping behavior recognition result.

5. The method according to claim 1, characterized in that, When the target detection result indicates that there is a handheld terminal in the image frame, the steps of performing associated recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result include: When there is a handheld terminal in the image frame, use the Hungarian algorithm to perform associated recognition on the safety officer and the handheld terminal in the target image frame with the handheld terminal to obtain a first associated recognition result. The associated recognition includes recognizing the relative position and relative distance between the safety officer and the handheld terminal; When the first associated recognition result indicates that there is an association between the safety officer and the handheld terminal, use the Hungarian algorithm to perform associated recognition on the handheld terminal and the hand key points of the safety officer to obtain a second associated recognition result; When the second associated recognition result indicates that there is an association between the handheld terminal and the hand key points of the safety officer, recognize the relative relationship between the handheld terminal and the target key points of the safety officer to obtain a third recognition result, where the target key points include at least one of the following: ear key points, eye key points.

6. The method according to claim 5, characterized in that, The steps of recognizing the relative relationship between the handheld terminal and the target key points of the safety officer include: Obtain the largest acute angle in the triangular relationship formed by the eye key points of the safety officer and the handheld terminal; Compare the angle value of the largest acute angle with a preset angle threshold to obtain a third comparison result; When the third comparison result indicates that the angle value of the largest acute angle is less than the preset angle threshold, determine that the safety officer has a first suspected terminal usage behavior, and obtain the target image frame in which the safety officer has the first suspected terminal usage behavior; Based on the adjacent image frames corresponding to the target image frame, perform tracking and recognition on the safety officer with the first suspected terminal usage behavior. When the safety officers in M adjacent image frames all have the first suspected terminal usage behavior, calculate the first terminal usage duration of the safety officer, where M is a positive integer; When the usage duration of the first terminal is greater than a preset terminal usage duration threshold, it is determined that the safety officer has the behavior of using the handheld terminal for entertainment.

7. The method according to claim 5, characterized in that, The step of identifying the relative relationship between the handheld terminal and the target key points of the safety officer further includes: Obtaining the distance value between the ear key point of the safety officer and the handheld terminal; Comparing the distance value with a preset distance threshold to obtain a fourth comparison result; When the fourth comparison result indicates that the distance value is less than the preset distance threshold, it is determined that the safety officer has a second suspected terminal usage behavior, and the target image frame when the safety officer has the second suspected terminal usage behavior is obtained; Based on the adjacent image frames corresponding to the target image frame, the safety officer with the second suspected terminal usage behavior is tracked and identified. When the safety officers in N adjacent image frames all have the second suspected terminal usage behavior, the second terminal usage duration of the safety officer is calculated, where N is a positive integer; When the second terminal usage duration is greater than a preset terminal usage duration threshold, it is determined that the safety officer has the behavior of using the handheld terminal to make a call.

8. The method according to claim 1, wherein The step of generating a dereliction of duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result further includes: When the third recognition result indicates that the safety officer has a handheld terminal usage behavior while on duty, the second recognition result is corrected based on the third recognition result to obtain the corrected second recognition result; Based on the first recognition result, the corrected second recognition result, and the third recognition result, the dereliction of duty behavior recognition result is generated.

9. An identification device for the dereliction of duty behavior of a safety officer, characterized in that, Includes: An acquisition unit for acquiring video data and preprocessing the video data to obtain a set of image frames to be recognized; A first recognition unit for performing target recognition on the image frames in the set of image frames to obtain a first recognition result, where the first recognition result is used to indicate whether the safety officer is on duty; A second recognition unit for, when the first recognition result indicates that the safety officer in the image frame is on duty, performing tracking and recognition on the safety officer in the image frame based on a target tracking algorithm to obtain a second recognition result, where the second recognition result is used to indicate whether the safety officer has the behavior of sleeping on duty; A detection unit for performing target detection on the set of image frames to obtain a target detection result, where the target detection result is used to indicate whether there is a handheld terminal in the image frame; A third recognition unit for, when the target detection result indicates that there is a handheld terminal in the image frame, performing association recognition and key point recognition on the safety officer and the handheld terminal to obtain a third recognition result, where the third recognition result is used to indicate whether the safety officer has a handheld terminal usage behavior; A generation unit for generating a dereliction of duty behavior recognition result based on the first recognition result, the second recognition result, and the third recognition result.

10. An electronic device, characterized in that, Comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for identifying the slacking behavior of a safety officer according to any one of claims 1 to 8.

Images