Method and device for detecting fatigue behavior of operator on duty and electronic equipment

Abstract

The invention discloses an on-duty personnel fatigue behavior detection method and device and electronic equipment, and belongs to the technical field of image detection, and the method comprises the steps: inputting a current frame image into an action detection model, and extracting feature embedded information and first label information; inputting a subject position label in the feature embedding information and the first label information into a target tracking model to obtain a subject identity label and a tracking position label; and inputting the current timestamp and second label information corresponding to the fatigue action label, the main body identity label and the tracking position label of the operator on duty into a post-processing model so as to enable the post-processing model to judge whether the second label information meets the fatigue behavior definition in a preset time period, and if so, outputting third label information and a fatigue alarm signal. According to the method, three-section detection of the action detection model, the target tracking model and the post-processing model is provided, and the time sequence modeling capability is only added into the post-processing model, so that the whole model is lighter, and the operation difficulty of fatigue behavior detection is further reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection, and more specifically, relates to a method, device and electronic equipment for detecting fatigue behavior of personnel on duty. Background Art

[0002] With the continuous advancement of the era of scientific and technological progress and technological development, an increasing number of large industrial scenes are gradually moving towards digital transformation, among which safe production is the most important link in the transformation of the industrial sector. For some industrial scenes prone to safety hazards, such as thermal power plants, sensors and other equipment are deployed and installed to record various parameters in the industrial production process in real time and transmit their data to office scenes such as the corresponding control room. Industrial scenes with safety hazards have very strict requirements for the safety of the production environment. It is required that there must be corresponding professionals in the control room to monitor various parameters in production in real time, so as to timely check possible safety hazards. This requires professionals in the control room to stay focused for a long time. At the same time, because early warning of safety hazards requires knowledge in multiple fields, and professionals are required to have certain data analysis and reasoning capabilities. Therefore, based on safety considerations, artificial intelligence cannot be used to completely replace these positions. However, due to the long-term high concentration of personnel, the physical and mental fatigue will lead to a decrease in reaction and prediction capabilities. If not discovered and handled in time, the inspection of safety hazards may be missed, threatening the safety of the production site. In recent years, with the continuous development of image processing technology, machine learning, and deep learning technology, the use of artificial intelligence technology to help detect personnel fatigue behavior has gradually replaced manual inspections.

[0003] The application areas of most existing fatigue behavior detection algorithms are concentrated on fatigue driving, and there are very few algorithms for non-face fatigue behavior detection. The difficulty of behavior detection is far greater than that of traditional target detection, and it is a very challenging topic in the field of computer vision. Unlike target recognition, behavior detection requires not only analyzing the spatial dependency of the target, but also analyzing the historical information of the target change, which increases the difficulty of behavior detection. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method, device and electronic equipment for detecting fatigue behavior of on-duty personnel, which aims to solve the technical problem that the existing deep learning-based limb behavior detection method is difficult to operate.

[0005] To achieve the above object, according to one aspect of the present invention, a method for detecting fatigue behavior of on-duty personnel is provided, comprising:

[0006] S1: Real-time acquisition of the current frame image of the on-duty personnel in the working environment;

[0007] S2: Input the current frame image into an action detection model including a feature extraction network and an action classification network, wherein the feature extraction network is used to extract key feature information from the current frame image, and the action classification network is used to extract feature embedding information and first label information of the on-duty personnel from the key feature information; the first label information includes: a fatigue action label and a subject position label;

[0008] S3: inputting the feature embedding information and the subject position label in the first label information of the on-duty personnel into a target tracking model to obtain a subject identity label and a tracking position label corresponding to the subject position label;

[0009] S4: Input the current timestamp and the second tag information corresponding to the fatigue action tag, subject identity tag, and tracking location tag of the on-duty personnel into the post-processing model, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second tag information of the corresponding on-duty personnel; the analyzer is used to determine whether the second tag information in the resource pool within a preset time period meets the fatigue behavior definition, and if so, outputs the corresponding third tag information and fatigue alarm signal of the on-duty personnel, the third tag information includes: fatigue behavior start and end time tags and second tag information.

[0010] In one embodiment, the feature extraction network in S2 includes:

[0011] An encoding module, used for extracting an initial feature map from the current frame image, and then encoding the initial feature map in combination with the subject position in the current frame image to obtain a spatial feature map sequence;

[0012] A decoding module is connected to the encoding module and the action classification network, and is used to extract the key feature information from the spatial feature map sequence and transmit the key feature information to the connected action classification network.

[0013] In one embodiment, the decoding module is a deformable attention module, comprising:

[0014] A sampling offset network, used to calculate the sampling offset of each feature vector of each spatial feature map in the sequence of spatial feature maps;

[0015] An attention weight allocation network, used to calculate the self-attention weight of each feature vector of each spatial feature map in the sequence of spatial feature maps;

[0016] The output network is used to fuse the sampling offset of each feature vector of each spatial feature map and the corresponding self-attention weight to obtain the key feature information.

[0017] In one of the embodiments, the decoding module is further used to generate an anchor query vector corresponding to the key feature information, and transmit the key feature information and its corresponding anchor query vector to the action classification network;

[0018] The elements of the anchor query vector include a query abscissa, a query ordinate, a query width and a query height.

[0019] In one embodiment, the target tracking model in S3 includes:

[0020] An extraction module, used for representing the feature embedding information as the main surface features of the on-duty personnel;

[0021] A matching module is connected to the matching module and is used to match the detected on-duty personnel with the on-duty personnel being tracked by using the subject appearance features and subject position labels of the on-duty personnel. If the match is successful, the detected on-duty personnel is assigned the same subject identity label as that of the on-duty personnel being tracked; wherein the on-duty personnel being tracked is the on-duty personnel detected in the previous frame image.

[0022] In one of the embodiments, the matching module is also used to, if the difference between the detected body appearance characteristics of the on-duty personnel and the body appearance characteristics of the on-duty personnel being tracked is within a first preset range, and the difference between the detected body location tag of the on-duty personnel and the body location tag of the on-duty personnel being tracked is within a second preset range, then the detected on-duty personnel is deemed to match the on-duty personnel being tracked.

[0023] In one of the embodiments, the analyzer is also used to, if it is detected that the resource pool continuously and continuously presents the second label information that meets the definition of fatigue behavior, it is deemed that the detected on-duty personnel are in a fatigue behavior state within the preset time period, and output continuous detection until the second label information of the on-duty personnel corresponds to a non-fatigue action, and output the corresponding third label information.

[0024] According to another aspect of the present invention, there is provided a device for detecting fatigue behavior of personnel on duty, comprising:

[0025] An acquisition module is used to acquire the current frame image of the on-duty personnel in the working environment in real time;

[0026] A detection module, configured to input the current frame image into a motion detection model including a feature extraction network and a motion classification network, wherein the feature extraction network is configured to extract key feature information from the current frame image, and the motion classification network is configured to extract feature embedding information and first label information of the on-duty personnel from the key feature information; the first label information includes: a fatigue motion label and a subject position label;

[0027] A tracking module, used for inputting the feature embedding information and the subject position label in the first label information of the on-duty personnel into a target tracking model to obtain a subject identity label and a tracking position label corresponding to the subject position label;

[0028] A post-processing module is used to input the current timestamp and the second label information corresponding to the fatigue action label, subject identity label, and tracking location label of the on-duty personnel into a post-processing model, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second label information of the corresponding on-duty personnel; the analyzer is used to determine whether the second label information in the resource pool within a preset time period meets the fatigue behavior definition, and if so, outputs the corresponding third label information and fatigue alarm signal of the on-duty personnel, the third label information includes: fatigue behavior start and end time labels and the second label information.

[0029] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0032] (1) The present invention provides a method for detecting fatigue behavior of on-duty personnel, comprising the steps of inputting the current frame image into an action detection model including a feature extraction network and an action classification network to extract feature embedding information and first label information of the on-duty personnel; inputting the feature embedding information and the subject position label in the first label information of the on-duty personnel into a target tracking model to obtain a subject identity label and a tracking position label corresponding to the subject position label; finally, inputting the current timestamp and the second label information corresponding to the fatigue action label, subject identity label, and tracking position label of the on-duty personnel into a post-processing model, so that the post-processing model determines whether the second label information meets the fatigue behavior definition within a preset time period, and if so, outputs the corresponding third label information of the on-duty personnel and a fatigue alarm signal. Most of the existing technologies couple the calculation of time series and spatial features, but the present invention proposes a three-stage fatigue behavior detection method. By utilizing the characteristic that fatigue behavior has small motion changes in the time dimension, a three-stage detection of motion detection model, target tracking model, and post-processing model is proposed. Each module is trained separately and functional decoupling is achieved. Different from the coupling of time series features and spatial features in conventional behavior detection, only the time series modeling capability is added to the post-processing model, making the overall model more lightweight, thereby reducing the operational difficulty of fatigue behavior detection.

[0033] (2) The encoding module of this scheme extracts an initial feature map from the current frame image, and then encodes the initial feature map in combination with the subject position in the current frame image to obtain a sequence of spatial feature maps; wherein the subject position in the current frame image takes into account the width and height information, which can be used to adjust the position attention feature map, thereby better capturing the spatial information of the target.

[0034] (3) The decoding module described in this scheme is a deformable attention module. The self-attention mechanism based on the sparsification strategy enables the motion detection model to pay more attention to the fatigue behavior of the on-duty personnel at fixed locations, making the motion detection model highly suitable for the fatigue behavior detection of on-duty personnel working at fixed locations.

[0035] (4) This scheme uses an explicitly represented anchor query vector instead of an implicitly represented target query vector, so that during the back-propagation process during training, the model only needs to transmit the gradient update of the coordinate position and label, rather than the gradient update of the entire feature map, which makes the model training converge faster.

[0036] (5) In this scheme, the feature embedding information generated by the self-attention mechanism based on the sparsification strategy of the extraction module replaces the feature map generated by the traditional convolutional neural network as the apparent feature of the subject, so that the apparent feature further weakens the background feature, highlights the regional features of the target subject, and improves the target tracking effect.

[0037] (6) This scheme uses the subject's appearance features combined with position information for matching, and combines motion and appearance descriptors to better avoid the frequent destruction and reconstruction of the target subject's identity label due to occlusion, making the entire tracking process more stable.

[0038] (7) The analyzer in this scheme can be used to process various timing-related detection tasks, including: continuous fatigue behavior detection where fatigue movements occur continuously over a period of time, periodic fatigue behavior detection where fatigue movements occur several times over a period of time, and one-time fatigue behavior detection where fatigue movements occur only once, making the detection tasks more diversified. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a method for detecting fatigue behavior of on-duty personnel provided by Embodiment 1 of the present invention;

[0040] Figure 2 Embodiment 1 of the present invention provides a framework diagram of an action detection model;

[0041] Figure 3 The first embodiment of the present invention provides an effect diagram of image enhancement preprocessing;

[0042] Figure 4 Embodiment 1 of the present invention provides a framework diagram of a target tracking model;

[0043] Figure 5 is a framework diagram of a post-processing module provided in Embodiment 1 of the present invention;

[0044] Figure 6 It is a framework diagram of another post-processing module provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment provides a method for detecting fatigue behavior of on-duty personnel, including: S1-S4. S1: Real-time acquisition of the current frame image of the on-duty personnel in the working environment. S2: Inputting the current frame image into an action detection model including a feature extraction network and an action classification network. The action detection model is as follows: Figure 2As shown, the feature extraction network is used to extract key feature information from the current frame image, and the action classification network is used to extract feature embedding information and the first label information of the on-duty personnel from the key feature information; the first label information includes: fatigue action label and subject position label. S3: Input the feature embedding information and the subject position label in the first label information of the on-duty personnel into the target tracking model to obtain the subject identity label and tracking position label corresponding to the subject position label. S4: Input the second label information corresponding to the fatigue action label, subject identity label, and tracking position label of the on-duty personnel of the current timestamp and the on-duty personnel, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second label information of the corresponding on-duty personnel; the analyzer is used to determine whether the second label information in the resource pool within a preset time period meets the fatigue behavior definition, and if so, outputs the corresponding third label information and fatigue alarm signal of the on-duty personnel, and the third label information includes: fatigue behavior start and end time labels and second label information.

[0048] Regarding S1, the collected real-time frame image is subjected to image enhancement preprocessing to obtain the current frame image, and the effect is as follows: Figure 3 The image enhancement preprocessing process is as follows: according to different duty scenes, the background part of the image is processed according to the weighted average grayscale processing method, and the three channel components of the image are weighted averaged with different weights. After weighted average of the three RGB components, a more reasonable grayscale image can be obtained. The image enhancement method can make the action detection model pay more attention to the fatigue behavior of people in specific positions, while ignoring the interference of non-target people in the background on the detection results.

[0049] Among them, the fatigue behavior definition comparison table can be as follows.

[0050] Table 1

[0051]

[0052]

[0053] The decoding module is connected to the encoding module and the action classification network, and is used to extract key feature information from the spatial feature map sequence and transmit the key feature information to the connected action classification network.

[0054] In one embodiment, the decoding module is a deformable attention module, including: a sampling offset network, an attention weight allocation network and an output network. The sampling offset network is used to calculate the sampling offset of each feature vector of each spatial feature map in the spatial feature map sequence; the attention weight allocation network is used to calculate the self-attention weight of each feature vector of each spatial feature map in the spatial feature map sequence; the output network is used to fuse the sampling offset of each feature vector of each spatial feature map and the corresponding self-attention weight to obtain key feature information.

[0055] When the existing Transformer is initialized, the attention weights assigned to all feature pixels are almost equal, which causes the action detection model to take a long time to learn to pay attention to truly meaningful positions, which should be sparse. At the same time, when the Transformer calculates the attention weights, it is accompanied by high computational complexity and spatial complexity. Especially in the encoding module, it is in a quadratic relationship with the number of feature pixels, so it is difficult to process high-resolution features. Most Transformer-type behavior detection models do not adopt a sparse strategy. In the encoder input, they directly process a continuous m-frame sequence f. i , i=1,2,…,m, and pass them through the feature extractor to obtain high-dimensional feature representations, and finally map them to the dimensional space of the encoder input through affine transformation, and expect that a randomly initialized token can learn the sum of the features of the historical frames under a specific behavior after the attention calculation of each layer of the encoder. Such a calculation process requires a huge amount of resources.

[0056] Therefore, it is necessary to adopt a sparse attention strategy. The decoding module in the feature extraction network uses a self-attention mechanism and introduces a deformable attention module. Analogous to the Deformable DETR model, it is inspired by sparse convolution and integrates the correlation modeling capabilities of sparse convolution and Transformer. Previously, for each frame query, it focused on the image features and spatial features of all other frames in the continuous frames as keys. However, for fatigue behaviors with small action changes and small optical flow changes, only a few key frames in the continuous frames are representative, and the remaining frames are highly redundant. Therefore, only focusing on more meaningful frames that the network believes contain more local information (a small and fixed number of positions as keys) can alleviate the problem of large computational complexity brought by the feature maps of continuous frames, thereby directly speeding up the inference speed of the action detection model. Under this decoding module, each feature pixel does not have to interact with all feature pixels for calculation, but only needs to interact with some other pixels obtained based on sampling, which greatly accelerates the convergence speed of the model and also reduces the computational complexity and required space resources. For sparse sampling within a single-frame image, the feature vector obtained at each point in the feature map is passed through the sampling offset network to obtain the N offset sampling points corresponding to the point. At the same time, the attention weight of each offset point when calculating self-attention is obtained through the attention weight allocation network. Finally, the result is mapped back to the dimension of the corresponding feature vector through the affine network. The sampling offset network, the attention weight allocation network, and the affine transformation network can all be obtained through learning.

[0057] The deformable attention module is an attention module used in the deformable DETR architecture, which attempts to overcome a problem with Transformer-based self-attention because it can look at all possible positions. Inspired by deformable convolutions, the deformable attention module only focuses on a small set of key sampling points around the reference point, regardless of the spatial size of the feature map. Each query in the traditional attention module calculates attention with all keys. In the deformable attention module, each query only samples the keys of some positions in the global position, and the value is also obtained by sampling and interpolation based on these positions. Finally, this local sparse attention weight is applied to the corresponding value, so the convergence time will be shortened.

[0058] In one embodiment, the decoding module is further used to generate an anchor query vector corresponding to the key feature information, and transmit the key feature information and its corresponding anchor query vector to the action classification network; the elements of the anchor query vector include the query horizontal coordinate, the query vertical coordinate, the query width and the query height. In the decoding module, an explicit and learnable query vector is used instead of an implicit expression. Each query vector can be represented as a four-tuple: (ψ i =(x (i) ,y(i) ,h (i) ,w (i) ), i = 1, ..., N); and only update Δx is calculated in the gradient update of the encoding module (i) ,Δy (i) ,Δh (i) ,Δw (i) In order to reduce the computational resources of the action detection model, a sparse strategy is used in the feature extraction network within a single frame image, and the displayed anchor query vectors are used as a substitute for object query vectors, making the model more lightweight and reducing the computational complexity of the action detection model inference.

[0059] In one embodiment, the target tracking model in S3 includes: an extraction module and a matching module. The extraction module is used to represent the feature embedding information as the main appearance features of the on-duty personnel; the matching module is connected to the matching module and is used to match the detected on-duty personnel with the on-duty personnel being tracked using the main appearance features and the main position labels of the on-duty personnel. If the match is successful, the detected on-duty personnel is given the same main identity label as the on-duty personnel being tracked; wherein the on-duty personnel being tracked are the on-duty personnel detected in the previous frame image.

[0060] In the target tracking stage, the target tracking network uses the Deepsort network and fine-tunes it. In the first step of target tracking, the feature extraction network is used to obtain the location box of the target to be detected and its action category. In the second step, an action predictor is used to use its past information to predict the location of its next action. Deepsort introduces deep learning into the SORT algorithm and reduces identity switching by adding appearance descriptors, making tracking more efficient. The appearance descriptor is the feature map of the detected target obtained by the feature extraction network in the previous step.

[0061] In the target estimation stage, the detection results are passed from the current frame to the next frame to estimate the position of the target in the next frame using Gaussian distribution and constant velocity model. The target estimation is estimated using a Kalman filter. In the data association stage, there are target bounding boxes and detected bounding boxes. Therefore, the cost matrix is ​​calculated as the intersection-over-union (IOU) distance between each detection box and all predicted bounding boxes of existing targets. In the stage of creating tracking target IDs, when an object enters or leaves the detection area, a unique object ID is created and destroyed accordingly. Deepsort uses a better association metric that combines motion and appearance descriptors. This can better avoid the frequent destruction and reconstruction of object IDs caused by occlusion, making the entire tracking process more stable. Among them, Deepsort can be defined as a tracking algorithm that tracks objects not only based on the speed and motion of the object, but also based on the appearance of the object. The target tracking model framework is as follows Figure 4 shown.

[0062] In one of the embodiments, the matching module is also used to, if the difference between the detected body appearance features of the on-duty person and the body appearance features of the on-duty person being tracked is within a first preset range, and the difference between the detected body position tag of the on-duty person and the body position tag of the on-duty person being tracked is within a second preset range, then the detected on-duty person is deemed to match the on-duty person being tracked.

[0063] In one of the embodiments, the analyzer is also used to, if it is detected that the resource pool continuously and continuously appears second label information that meets the definition of fatigue behavior, it is deemed that the detected on-duty personnel are in a fatigue behavior state within a preset time period, and the analyzer outputs continuous detection until the second label information of the on-duty personnel corresponds to a non-fatigue action, and outputs the corresponding third label information.

[0064] The post-processing process is the key stage of fatigue behavior detection. This paper proposes post-processing logic to add timing information to static labels. When the model reasoning process reaches the model tracking stage, it will target frame l i Generate the detection label Φ under this frame i , where the label Φ i Including the position coordinates of the detected object, the formula is as follows:

[0065]

[0066] Where m means that m fatigue motion objects are detected in this frame, x k Represents the normalized first dimension coordinate of the upper left corner of the kth fatigue object detection box, y k represents the normalized second dimension coordinate of the upper left corner of the kth fatigue object detection box, h k represents the normalized height of the kth fatigue object detection box, w k Represents the normalized width of the k-th fatigue detection object detection box.

[0067] TagsΦ i It also includes the fatigue action label of the detected object, and the formula is as follows:

[0068]

[0069] TagsΦ i It also includes the person ID of the detected object. The formula is as follows:

[0070] (I k ,k=1,…,m);

[0071] TagsΦ i Also includes the timestamp of the detected frame: t i , so:

[0072]

[0073] The post-processing process can be viewed as a multi-threaded finite state machine, which creates different behavior detection threads for different tracked objects and outputs labels Φ for tracking at most N frames. i , i=1,…,N establishes a circular queue resource pool T to store the fatigue behavior detected in the recent time, and different threads use synchronization locks to mutually exclusive access to resource pool T.

[0074] Thread creation and destruction correspond to different states in the finite state machine. Threads are in the stopped state at the beginning of the detection. When fatigue action appears in the tracking network output label It is regarded as triggering the state transition logic, and the finite state machine enters the next state and changes the state according to the corresponding character label I k Create the corresponding fatigue behavior detection thread. Different threads correspond to their unique finite state machine states, and different thread states do not interfere with each other. When a fatigue action label that meets the fatigue behavior definition of the same object and has a certain continuous time interval appears in the resource pool. We can indirectly assume that the object is in the fatigue behavior state during this time interval until the resource pool sees a transition in the fatigue action of the object: the action changes from fatigue action to non-fatigue action. The finite state machine state under the thread changes again, the state machine terminates, the thread is destroyed, and the fatigue behavior label is recorded: Where T k Indicates the duration of the fatigue behavior: In this way, the timing information of fatigue actions is indirectly obtained, a lightweight strategy is implemented, the reasoning time is reduced, the model size is reduced, and a fatigue behavior detection algorithm suitable for real environments is realized.

[0075] The framework for detecting fatigue behavior and alarming is as follows Figure 5 As shown. First, a continuous video frame is obtained through the video stream of the monitoring point, and the feature target frame that may exist in the scene is obtained through the action detection model. Then, the continuous tracking of the target body is realized through the multi-target tracking model. For fatigue behavior detection that meets the continuous characteristics, the detection results are put into the post-processing timing analysis model to build timing information for each target body. If the detection label of the target body to be detected in the buffer of the post-processing model meets the definition of the fatigue behavior: the action under the behavior occurs continuously within a period of time, then it is considered that the corresponding fatigue behavior has occurred within the time period. Finally, if the corresponding target body has fatigue behavior, an alarm video is generated according to the detection results taken out of the buffer and an alarm is sent.

[0076] In addition, if the fatigue behavior to be detected is relatively complex, a fixed-length continuous frame post-processing model can be used to model its time series. The framework for detecting fatigue behavior and alarming is as follows: Figure 6As shown in Figure 1. In this process, a continuous multi-frame sequence of the same object is used as input. After each frame image passes through the feature extractor, sparse sampling between multiple continuous frames is used, so that each frame image selectively performs self-attention calculation with other frame images, thereby further reducing the amount of inference calculation. The task token is a randomly initialized, learnable feature vector, whose purpose is to integrate the historical frame f -T →The image area features related to fatigue behavior in f0. After passing through the encoder, the final behavior feature vector is obtained, and finally the behavior label of the fatigue behavior is obtained by using the vector through a linear classifier. Similarly, if the corresponding target body has fatigue behavior, an alarm video is generated based on the detection results taken out of the buffer and an alarm is sent.

[0077] Example 2

[0078] This embodiment provides a device for detecting fatigue behavior of on-duty personnel, including: an acquisition module, a detection module, a tracking module and a post-processing module. Among them, the acquisition module is used to acquire the current frame image of the on-duty personnel in the working environment in real time; the detection module is used to input the current frame image into the action detection model including the feature extraction network and the action classification network, the feature extraction network is used to extract key feature information from the current frame image, and the action classification network is used to extract feature embedding information and the first label information of the on-duty personnel from the key feature information; the first label information includes: fatigue action label and subject position label; the tracking module is used to input the feature embedding information and the subject position label in the first label information of the on-duty personnel into the target tracking model, and obtain the subject identity label and tracking position label corresponding to the subject position label; the post-processing module is used to input the current timestamp and the fatigue action label, subject identity label, and tracking position label of the on-duty personnel The corresponding second label information is input into the post-processing model, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second label information of the corresponding on-duty personnel; the analyzer is used to determine whether the second label information in the resource pool within a preset time period meets the fatigue behavior definition, and if it does, outputs the corresponding third label information and fatigue alarm signal of the on-duty personnel, and the third label information includes: fatigue behavior start and end time labels and second label information.

[0079] Example 3

[0080] This embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the steps of the method are implemented when the processor executes the computer program.

[0081] Example 4

[0082] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0083] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting fatigue behavior of on-duty personnel, characterized in that: include: S1: Real-time acquisition of the current frame image of the on-duty personnel in the working environment; S2: Input the current frame image into an action detection model including a feature extraction network and an action classification network, wherein the feature extraction network is used to extract key feature information from the current frame image, and the action classification network is used to extract feature embedding information and first label information of the on-duty personnel from the key feature information; the first label information includes: a fatigue action label and a subject position label; S3: inputting the feature embedding information and the subject position label in the first label information of the on-duty personnel into a target tracking model to obtain a subject identity label and a tracking position label corresponding to the subject position label; S4: Input the current timestamp and the second tag information corresponding to the fatigue action tag, subject identity tag, and tracking location tag of the on-duty personnel into the post-processing model, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second tag information of the corresponding on-duty personnel; the analyzer is used to determine whether the second tag information in the resource pool within a preset time period meets the fatigue behavior definition, and if so, outputs the corresponding third tag information and fatigue alarm signal of the on-duty personnel, the third tag information includes: fatigue behavior start and end time tags and second tag information.

2. The method for detecting fatigue behavior of on-duty personnel according to claim 1, characterized in that: The feature extraction network in S2 includes: An encoding module, used to extract an initial feature map from the current frame image, and then encode the initial feature map in combination with the subject position in the current frame image to obtain a spatial feature map sequence; A decoding module is connected to the encoding module and the action classification network, and is used to extract the key feature information from the spatial feature map sequence and transmit the key feature information to the connected action classification network.

3. The method for detecting fatigue behavior of on-duty personnel according to claim 2, characterized in that: The decoding module is a deformable attention module, comprising: A sampling offset network, used to calculate the sampling offset of each feature vector of each spatial feature map in the sequence of spatial feature maps; An attention weight allocation network, used to calculate the self-attention weight of each feature vector of each spatial feature map in the sequence of spatial feature maps; The output network is used to fuse the sampling offset of each feature vector of each spatial feature map and the corresponding self-attention weight to obtain the key feature information.

4. The method for detecting fatigue behavior of on-duty personnel according to claim 2, characterized in that: The decoding module is also used to generate an anchor query vector corresponding to the key feature information, and transmit the key feature information and its corresponding anchor query vector to the action classification network; The elements of the anchor query vector include a query abscissa, a query ordinate, a query width and a query height.

5. The method for detecting fatigue behavior of on-duty personnel according to claim 1, characterized in that: The target tracking model in S3 includes: An extraction module, used for representing the feature embedding information as the main surface features of the on-duty personnel; A matching module is connected to the matching module and is used to match the detected on-duty personnel with the on-duty personnel being tracked by using the subject appearance features and subject position labels of the on-duty personnel. If the match is successful, the detected on-duty personnel is assigned the same subject identity label as that of the on-duty personnel being tracked; wherein the on-duty personnel being tracked is the on-duty personnel detected in the previous frame image.

6. The method for detecting fatigue behavior of on-duty personnel according to claim 5, characterized in that: The matching module is also used to, if the difference between the detected main appearance characteristics of the on-duty personnel and the main appearance characteristics of the on-duty personnel being tracked is within a first preset range, and the difference between the detected main location tag of the on-duty personnel and the main location tag of the on-duty personnel being tracked is within a second preset range, then the detected on-duty personnel is deemed to match the on-duty personnel being tracked.

7. The method for detecting fatigue behavior of on-duty personnel according to any one of claims 1 to 6, characterized in that: The analyzer is also used to, if it is detected that the resource pool continuously and continuously presents the second label information that meets the definition of fatigue behavior, it is deemed that the detected on-duty personnel are in a fatigue behavior state within the preset time period, and output continuous detection until the second label information of the on-duty personnel corresponds to a non-fatigue action, and output the corresponding third label information.

8. A device for detecting fatigue behavior of on-duty personnel, characterized in that: include: An acquisition module is used to acquire the current frame image of the on-duty personnel in the working environment in real time; A detection module, configured to input the current frame image into an action detection model including a feature extraction network and an action classification network, wherein the feature extraction network is configured to extract key feature information from the current frame image, and the action classification network is configured to extract feature embedding information and first label information of the on-duty personnel from the key feature information; The first tag information includes: fatigue action tag and subject position tag; A tracking module, used for inputting the feature embedding information and the subject position label in the first label information of the on-duty personnel into a target tracking model to obtain a subject identity label and a tracking position label corresponding to the subject position label; A post-processing module is used to input the current timestamp and the second label information corresponding to the fatigue action label, subject identity label, and tracking location label of the on-duty personnel into a post-processing model, so that the post-processing model establishes a corresponding analysis thread for each on-duty personnel; the analysis thread includes: a resource pool and an analyzer, the resource pool is used to store the second label information of the corresponding on-duty personnel; the analyzer is used to determine whether the second label information in the resource pool within a preset time period meets the fatigue behavior definition, and if so, outputs the corresponding third label information and fatigue alarm signal of the on-duty personnel, the third label information includes: fatigue behavior start and end time labels and the second label information.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images