Personnel status detection methods, electronic devices, storage media and software products

Abstract

This application discloses a method, electronic device, storage medium, and program product for detecting human state, relating to the field of computer vision technology. Through a pre-trained sleep state detection model, it can efficiently and on a large scale extract the spatial position features and temporal features of displacement changes of the head, limbs, and torso of a person. It can cover three typical sleep states: "nodding, sleeping face down, and sleeping on your back," avoiding missed detections caused by single features. Furthermore, based on spatiotemporal dual-dimensional feature judgment, it can effectively distinguish between "sleep" and "normal state," avoiding the subjectivity of manual sleep detection and improving the accuracy of detecting human sleep state.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for detecting personnel status, an electronic device, a storage medium, and a program product. Background Technology

[0002] Positions such as park security guards, traffic accident investigators, and medical ward monitors require long hours of static duty and high vigilance to prevent safety incidents. For example, security guards monitor park dynamics, traffic officers detect traffic risks, and monitor patients' vital signs. However, the monotonous work content and environment can easily lead to physical and mental fatigue among staff, especially at night or in the early morning when they are more likely to fall asleep. This can not only lead to dereliction of duty but may also cause serious safety hazards such as park security incidents, traffic accidents, and delays in patient treatment. Therefore, it is necessary to conduct real-time sleep monitoring on personnel to avoid safety risks.

[0003] In related technologies, videos of people are manually observed to identify whether they are asleep. However, manual sleep detection is highly subjective, making it difficult to accurately identify sleep states, and it is also difficult to perform sleep state detection on a large number of people simultaneously. Summary of the Invention

[0004] The personnel status detection method, electronic device, storage medium, and program product provided in this application at least solve the problems of strong subjectivity, difficulty in accurately identifying sleep status, and difficulty in simultaneously detecting the sleep status of a large number of people in related technologies.

[0005] This application provides a method for detecting personnel status, including:

[0006] Obtain video data of personnel;

[0007] The video data is input into the trained sleep state detection model, which extracts and identifies the spatial position features of the head, limbs and torso of the person from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, and outputs the person's head nodding frequency information, prone sleeping duration information and supine sleeping duration information.

[0008] If the head nodding frequency information is determined to be greater than the first preset threshold within a preset time, the prone sleeping time information is greater than the second preset threshold, and / or the lying down sleeping time information is greater than the third preset threshold, then the person will be marked as being in a sleep state.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described personnel status detection methods.

[0010] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described personnel status detection methods.

[0011] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described personnel status detection methods.

[0012] This application utilizes a pre-trained sleep state detection model to efficiently and on a large scale extract spatial position features and temporal features of displacement changes in the head, limbs, and torso of individuals. It covers three typical sleep states: "nodding, prone sleeping, and lying down sleeping," avoiding missed detections caused by single features. Furthermore, based on spatiotemporal dual-dimensional feature judgment, it can effectively distinguish between "sleep" and "normal state," avoiding the subjectivity of manual sleep detection and improving the accuracy of sleep state detection. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 Flowchart of the personnel status detection method provided in the embodiments of this application Figure 1 ;

[0015] Figure 2 Flowchart of the personnel status detection method provided in the embodiments of this application Figure 2 ;

[0016] Figure 3 A schematic diagram of the architecture of the sleep state detection model provided in the embodiments of this application;

[0017] Figure 4 This is a schematic diagram of the feature fusion layer architecture provided in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of the personnel status detection device provided in the embodiments of this application;

[0019] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To address the aforementioned technical challenges of subjective and inaccurate sleep identification in manual sleep detection, as well as the difficulty in simultaneously detecting sleep patterns in large numbers of individuals, the following technical concept is proposed: A pre-trained language model can extract spatial and temporal features of individuals from video data, transforming the subjective behavior of sleep into quantifiable metrics. Objective metrics can then be used to determine sleep status, replacing subjective judgment and eliminating human error. Furthermore, the automated pre-trained language model can efficiently process large amounts of video data in parallel, offering higher efficiency compared to manual methods.

[0023] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 Flowchart of the personnel status detection method provided in the embodiments of this application Figure 1 ,like Figure 1 As shown, the personnel status detection method provided in the embodiments of this application can be applied to any electronic device. The method is described in detail below:

[0025] S101. Obtain video data of personnel.

[0026] It is possible to obtain personnel video data from lower-level video systems, and the video data must meet the preset data format.

[0027] S102. Input the video data into the trained sleep state detection model, so that the sleep state detection model can extract and identify the spatial position features of the person's head, limbs and torso from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, and output the person's nodding frequency information, prone sleeping duration information and supine sleeping duration information.

[0028] The sleep state detection model determines head nodding frequency by measuring the head's displacement relative to the torso or limbs. A head nod is defined as one occurrence when the displacement relative to the torso or limb is less than a preset displacement threshold in the direction between the head and the corresponding torso or limb. Specifically, the formula for calculating head nodding frequency is as follows:

[0029]

[0030]

[0031] in, This provides the head-nodding frequency information for the i-th window. For a sliding time window, the time range of the i-th window is... , Let k be the time when the nod occurs. Let N be the number of nods contained in the i-th window, and N be the total number of nods.

[0032] S103. If it is determined that the nodding frequency information is greater than the first preset threshold, the prone sleeping duration information is greater than the second preset threshold, and / or the lying down sleeping duration information is greater than the third preset threshold within a preset time, then the person is marked as being in a sleep state.

[0033] For example, nodding 7 times within 1 minute exceeds the first preset threshold of 5 times, triggering a judgment. Sleeping face down for 40 seconds exceeds the second preset threshold of 30 seconds, triggering a judgment. Sleeping lying down for 1 minute and 20 seconds exceeds the third preset threshold of 1 minute, triggering a judgment.

[0034] The personnel state detection method provided in this application, through a pre-trained sleep state detection model, can efficiently and on a large scale extract the spatial position features and temporal features of displacement changes of the head, limbs and torso of personnel. It can cover three typical sleep states: "nodding, sleeping face down and sleeping on your back", avoiding missed detections caused by single features. Moreover, based on spatiotemporal dual-dimensional feature judgment, it can effectively distinguish between "sleep" and "normal state", avoiding the subjectivity of manual sleep detection and improving the accuracy of personnel sleep state detection.

[0035] Figure 2 Flowchart of the personnel status detection method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, the method includes:

[0036] S201. Obtain video data of personnel.

[0037] In one possible implementation, raw video data of personnel is collected. The raw video data is preprocessed to obtain preprocessed video data. The maximum frame rate information that the sleep state detection model can process is obtained. If the frame rate information of the preprocessed video data is greater than the maximum processable frame rate information, frames are discarded from the preprocessed video data so that the frame rate information of the preprocessed video data is less than or equal to the maximum processable frame rate information, thus obtaining the personnel's video data. Otherwise, the preprocessed video data is used as the personnel's video data.

[0038] The frame rate relationship between personnel video data, pre-processed video data, and the maximum processable frame rate information is shown in the following formula:

[0039]

[0040] in, For the frame rate information of the video data of personnel, For preprocessed video data frame rate information, This represents the maximum frame rate information that can be processed.

[0041] Specifically, frame dropping is performed on the preprocessed video data to make the frame rate information of the preprocessed video data less than or equal to the maximum processable frame rate information, in order to obtain the video data of the personnel, including:

[0042] S2011. Decode the preprocessed video data into image data.

[0043] S2012. Sample and discard redundant video frames from the image data to obtain the image data with the dropped frames. For example, if the maximum processable frame rate is 20 frames and the preprocessed video data frame rate is 25 frames, sample the image data according to the frequency and discard the redundant 5 frames.

[0044] S2013. Re-encode the lost frame image data into video data to obtain the video data of the personnel.

[0045] In one possible implementation, each frame of image data in the original video book data is processed to a preset size and then filled with grayscale to obtain video data of the preset size; a preset filtering algorithm is used to filter noise in the video data of the preset size to obtain denoised video data; a preset motion blur removal algorithm is used to remove motion blur in the denoised video data to obtain preprocessed video data.

[0046] For example, each frame of image data in the original video data is processed to a preset size, which is 640*640. If the frame image data is not square, the largest side of the frame image data is scaled to 640, and the smaller side is filled with grayscale to 640. The preset filtering algorithm can be Gaussian filtering, bilateral filtering, or mean filtering, etc. The preset deblurring algorithm can be inverse filtering, Wiener filtering, or blind deblurring, etc.

[0047] In one possible implementation, prior to S202, the following is included:

[0048] Construct an initial sleep state detection model. Obtain training data. Construct a total loss function based on preset head nodding frequency loss functions, preset prone sleeping loss functions, and preset supine sleeping loss functions. Based on the total loss function, train the initial sleep state detection model using the training data to obtain a trained sleep state detection model.

[0049] Optionally, an initial sleep state detection model can be constructed based on a 3D convolutional neural network.

[0050] In one possible implementation, a preset head-nodding frequency loss function is included, comprising:

[0051]

[0052] in, The preset head-nodding frequency loss function is used. For real labels, For predicting labels, This is the first hyperparameter.

[0053] In one possible implementation, a preset prone sleeping loss function is included, comprising:

[0054]

[0055] in, To pre-define the prostrate sleeping loss function, For real labels, For predicting labels, This is the second hyperparameter.

[0056] In one possible implementation, a pre-defined sleep loss function is included, comprising:

[0057]

[0058] in, To preset the sleep loss function, For real labels, For predicting labels, This is the third hyperparameter.

[0059] In one possible implementation, the total loss function includes:

[0060]

[0061] in, For the total loss function, The preset head-nodding frequency loss function is used. To pre-define the prostrate sleeping loss function, To preset the sleep loss function, , and These are the fourth, fifth, and sixth hyperparameters, respectively.

[0062] S202. Input the video data into the trained sleep state detection model, so that the sleep state detection model can extract and identify the spatial position features of the person's head, limbs and torso from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, and output the person's nodding frequency information, prone sleeping duration information and supine sleeping duration information.

[0063] In one possible implementation, Figure 3 This is a schematic diagram of the architecture of the sleep state detection model provided in the embodiments of this application. Figure 3 Based on the sleep state detection model shown, S202 includes the following steps:

[0064] S2021. Using the first feature extraction layer, the spatial position features of the head, limbs and torso of the person are extracted from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, to obtain the first spatiotemporal feature data.

[0065] For example, the first feature extraction layer can be a three-dimensional convolutional layer (3DConvolution, 3DConv) with a kernel size of 7X7X7. The 3DConv layer includes three-dimensional convolution (Convolution3d, Conv3d), batch normalization (BN) and sigmoid linear unit (SiLU).

[0066] S2022. Multiple pooling layers are used to pool the first spatiotemporal feature data to obtain the second spatiotemporal feature data.

[0067] S2023. Using a second feature extraction layer and a third feature extraction layer, the core features of the second spatiotemporal feature data are further extracted to obtain the third spatiotemporal feature data.

[0068] For example, the second feature extraction layer can be a 3DConv layer with a kernel size of 1X1X1, and the third feature extraction layer can be a 3DConv layer with a kernel size of 3X3X3.

[0069] S2024. Multiple pooling layers are used to pool the third spatiotemporal feature data to obtain the fourth spatiotemporal feature data.

[0070] S2025. A feature fusion layer is adopted to extract and fuse shallow detail features, deep local features and global structural features from the fourth spatiotemporal feature data to obtain the fifth spatiotemporal feature data.

[0071] It is possible that in step S2025, multiple feature fusion layers can be used for feature fusion processing to improve the accuracy of feature fusion.

[0072] S2026. An upsampling layer is used to process the feature dimensions of the fifth spatiotemporal feature data to be the same as those of the third spatiotemporal feature data, so as to obtain the sixth spatiotemporal feature data.

[0073] S2027. Merge the sixth spatiotemporal feature data with the third spatiotemporal feature data to obtain the seventh spatiotemporal feature data.

[0074] S2028. A feature fusion layer is used to extract and fuse shallow detail features, deep local features, and global structural features from the seventh spatiotemporal feature data to obtain the eighth spatiotemporal feature data.

[0075] It is possible that in step S2028, multiple feature fusion layers can be used for feature fusion processing to improve the accuracy of feature fusion.

[0076] S2029. Multiple pooling layers are used to pool the eighth spatiotemporal feature data to obtain the ninth spatiotemporal feature data.

[0077] S20210. A feature fusion layer is used to extract and fuse shallow detail features, deep local features, and global structural features from the ninth spatiotemporal feature data to obtain the tenth spatiotemporal feature data.

[0078] S20211. Using the third feature extraction layer and the second feature extraction layer, the core features of the tenth spatiotemporal feature data are further extracted to obtain the eleventh spatiotemporal feature data.

[0079] S20212. Input the eleventh spatiotemporal feature data into the inference layer to obtain information on the frequency of head nodding, duration of prone sleeping, and duration of lying down sleeping.

[0080] In one possible implementation, the plurality of pooling layers includes one or more of a first max pooling layer, a second max pooling layer, and an average pooling layer.

[0081] For example, such as Figure 3 As shown, the multiple pooling layers include a first max pooling layer, a second max pooling layer, and an average pooling layer in sequence. The pooling window of the first max pooling layer is 1x3x3, and the stride is 1x2x2. The pooling window of the second max pooling layer is 3x3x3, and the stride is 2x2x2. The pooling window of the average pooling layer is 2x7x7, and the stride is 2x2x2.

[0082] In one possible implementation, the feature fusion layer includes a second feature extraction layer, a third feature extraction layer, a feature segmentation layer, and a fusion layer, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the feature fusion layer architecture provided in an embodiment of this application. Step S2025 specifically includes the following steps:

[0083] S20251. Using a second feature extraction layer and a feature segmentation layer, shallow detail features are extracted from the fourth spatiotemporal feature data and feature segmentation is performed to obtain the first shallow detail features and the second shallow detail features.

[0084] Specifically, after processing by the second feature extraction layer, the fourth spatiotemporal feature data is then segmented into C / 2 and C / 2 in the channel dimension by the feature segmentation layer to obtain the first shallow detail feature and the second shallow detail feature.

[0085] S20252. Using a second feature extraction layer and a third feature extraction layer, deep feature extraction is performed on the first shallow layer detail features to obtain deep local features.

[0086] S20253. Using the third feature extraction layer and the second feature extraction layer, global structural features are extracted from the fourth spatiotemporal feature data.

[0087] S20254 employs a fusion layer that integrates first shallow detail features, second shallow detail features, deep local features, and global structural features to obtain fused feature data.

[0088] S20255. The second feature extraction layer is used to extract features from the fused feature data to obtain the fifth spatiotemporal feature data.

[0089] It should be noted that the specific implementation methods of steps S2028 and S20210 can be referred to the specific implementation methods of steps S20251 to S20255, and will not be elaborated further here.

[0090] S203. If it is determined that the nodding frequency information is greater than the first preset threshold, the prone sleeping duration information is greater than the second preset threshold, and / or the lying down sleeping duration information is greater than the third preset threshold within a preset time, then the person is marked as being in a sleep state.

[0091] S204. If a person is marked as being in a sleep state, a sleep reminder message is sent to the person's terminal device, and the duration of the person's sleep state is continuously recorded.

[0092] For example, a sleep reminder message could be "Sleep mode detected, please focus on your work".

[0093] S205. If the duration of sleep exceeds the fourth preset threshold, a sleep alarm message is sent to the terminal device of the person's superior.

[0094] For example, a sleep alarm message could be: "Employee XXX has been asleep for more than 10 minutes. Please intervene."

[0095] S206. If the duration of sleep exceeds the fifth preset threshold, a sleep alarm message is sent to the terminal device of the person's superior management personnel.

[0096] For example, a sleep alarm message could be: "Employee XXX has been asleep for more than 30 minutes. The supervisor has not dealt with it in time. Please coordinate urgently."

[0097] In one possible implementation, the internal temperature of the electronic device to which the personnel status detection method provided in this embodiment is applied is continuously monitored. If the internal temperature of the device is greater than or equal to a preset temperature threshold, the sleep detection process of the operating personnel is stopped until the internal temperature of the device is less than the preset temperature threshold, and then the previous process is continued. This is to prevent the temperature from being too high during the operation of the device, which would cause the image processor and central processing unit to reduce their frequency, reduce the frame rate of video parsing, prevent full video parsing, and cause video blocking.

[0098] The personnel status detection method provided in this application, through a pre-trained sleep state detection model, can efficiently and on a large scale extract the spatial position features and temporal features of displacement changes of the head, limbs, and torso of personnel. It covers three typical sleep states: nodding, prone sleeping, and lying down, avoiding missed detections due to single features. Furthermore, based on spatiotemporal dual-dimensional feature judgment, it can effectively distinguish between "sleep" and "normal state," avoiding the subjectivity of manual sleep detection and improving the accuracy of personnel sleep state detection. Simultaneously, the hierarchical progressive alarm system of "sleeping personnel → superior management → higher-level management" realizes a scientific intervention logic of "self-correction first, then step-by-step supervision." First, a reminder is sent to the personnel, giving them an opportunity to wake up independently, avoiding unnecessary management intervention. Only when self-correction is ineffective or the sleep problem is severe are superior and higher-level management personnel introduced, ensuring precise and efficient intervention and reducing the ineffective consumption of management resources.

[0099] By obtaining the maximum processable frame rate of the sleep state detection model and adjusting the frame rate of the preprocessed video data accordingly, it ensures that the frame rate of the video data input to the model is always within the model's processing capacity. This avoids the sleep state detection model from experiencing lag, crashes, or parsing failures due to "high frame rate data exceeding the model's capacity limit," ensuring the continuous and stable operation of the detection process and adapting to deployment devices with different computing power performance.

[0100] Standardized data format ensures consistent model input: By processing each frame of the original video to a preset size and padding the short sides with grayscale, it avoids input size inconsistencies caused by differences in original video resolution, providing a standardized data foundation for subsequent feature extraction and improving data compatibility across different video devices and scenarios. Noise interference is filtered to improve feature extraction accuracy. Motion blur is eliminated to restore key dynamic features.

[0101] By employing a progressive structure of "first feature extraction layer → multi-pooling layer → second / third feature extraction layer," the spatial and temporal features of the head, limbs, and torso are first captured completely from the video. Then, multiple rounds of pooling processing are used to filter key features and remove redundant information. Finally, subsequent feature extraction layers refine the core features. This layered extraction mode avoids the loss of details in shallow features and prevents semantic ambiguity in deep features, providing a comprehensive and accurate spatiotemporal feature foundation for sleep behavior recognition.

[0102] The fourth, seventh, and ninth spatiotemporal feature data are processed by three feature fusion layers respectively, and "shallow detail features, deep local features, and global structural features" are fused in a targeted manner to achieve complementary advantages of features at different levels. At the same time, the fifth spatiotemporal feature data dimension is adjusted to be consistent with the third spatiotemporal feature data by using an upsampling layer, which solves the problem of size mismatch between deep features and shallow features. This ensures that the merged seventh spatiotemporal feature data retains both shallow detail accuracy and deep semantic depth, effectively reducing "misjudgment of similar states" and "missed detection of small actions".

[0103] Pooling operations using multiple pooling layers can effectively prevent rapid changes in the time dimension and prevent the mixing of edge features from different times in shallow networks, which would affect the extraction of spatial features.

[0104] The feature fusion layer extracts and segments the first and second shallow detail features from the fourth spatiotemporal feature data through a "second feature extraction layer + feature segmentation layer," achieving accurate segmentation of basic details such as "head edge texture and minor limb displacements." Then, the first shallow detail features are further processed through a "second + third feature extraction layer," resulting in deep local features focusing on "head nodding frequency and changes in facial occlusion during prone sleeping." Simultaneously, global structural features are extracted independently. This layered design of "shallow segmentation - deep deepening - global independence" significantly improves the recognizability of features at different levels, avoiding the problems of "local details being masked by global information" or "global semantics being interfered with by detail noise" caused by feature mixing.

[0105] Instead of using a single loss function, we designed pre-defined loss functions for three core detection tasks: head nodding frequency, prone sleeping, and supine sleeping. Different loss functions can accurately optimize the model parameters for the corresponding tasks. The head nodding frequency loss function can strengthen the learning of the periodicity of head displacement, while the prone sleeping loss function can focus on optimizing the recognition weight of the torso state in occluded scenes. This avoids "undertraining of some tasks" due to task differences, and ultimately achieves a simultaneous improvement in the accuracy of the three types of sleep behavior detection.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0107] Figure 5 This is a schematic diagram of the personnel status detection device provided in an embodiment of this application. Figure 5 As shown, embodiments of this application also provide a personnel status detection device 50, including an acquisition module 501, a processing module 502, and a determination module 503.

[0108] Module 501 is used to acquire video data of personnel;

[0109] The processing module 502 is used to input video data into the trained sleep state detection model, so that the sleep state detection model can extract and identify the spatial position features of the person's head, limbs and torso from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, and output the person's nodding frequency information, prone sleeping duration information and supine sleeping duration information.

[0110] The determination module 503 is used to mark the person as asleep if the head nodding frequency information is greater than a first preset threshold, the prone sleeping duration information is greater than a second preset threshold, and / or the lying sleeping duration information is greater than a third preset threshold within a preset time.

[0111] In one possible implementation, the acquisition module 501 is specifically used for:

[0112] The raw video data collected by the personnel;

[0113] The raw video data is preprocessed to obtain preprocessed video data;

[0114] Obtain the maximum frame rate information that the sleep state detection model can process;

[0115] If the frame rate of the preprocessed video data is greater than the maximum processable frame rate, then the preprocessed video data is subjected to frame discarding, so that the frame rate of the preprocessed video data is less than or equal to the maximum processable frame rate, in order to obtain the video data of the person.

[0116] Otherwise, the preprocessed video data will be used as the personnel's video data.

[0117] In one possible implementation, the acquisition module 501 is specifically used for:

[0118] Each frame of image data in the original video data is processed to a preset size and then filled with grayscale to obtain video data of the preset size;

[0119] A preset filtering algorithm is used to filter noise from video data of a preset size to obtain denoised video data;

[0120] A preset motion blur removal algorithm is used to remove motion blur from the denoised video data to obtain preprocessed video data.

[0121] In one possible implementation, the determination module 503 is specifically used for:

[0122] If a person is marked as asleep, a sleep reminder message will be sent to the person's terminal device, and the duration of the person's sleep state will be continuously recorded.

[0123] If the duration of sleep exceeds the fourth preset threshold, a sleep alarm message will be sent to the terminal device of the person's superior.

[0124] If the duration of sleep exceeds the fifth preset threshold, a sleep alarm will be sent to the terminal device of the person's superior supervisor.

[0125] In one possible implementation, the processing module 502 is specifically used for:

[0126] The first feature extraction layer is used to extract the spatial position features of the head, limbs and torso of the person from the video data, as well as the temporal features of the displacement changes of the head, limbs and torso, to obtain the first spatiotemporal feature data.

[0127] Multiple pooling layers are used to pool the first spatiotemporal feature data to obtain the second spatiotemporal feature data;

[0128] A second feature extraction layer and a third feature extraction layer are used to further extract core features from the second spatiotemporal feature data to obtain the third spatiotemporal feature data;

[0129] Multiple pooling layers are used to pool the third spatiotemporal feature data to obtain the fourth spatiotemporal feature data;

[0130] A feature fusion layer is used to extract and fuse shallow detail features, deep local features and global structural features from the fourth spatiotemporal feature data to obtain the fifth spatiotemporal feature data.

[0131] An upsampling layer is used to process the feature dimensions of the fifth spatiotemporal feature data to be the same as those of the third spatiotemporal feature data, so as to obtain the sixth spatiotemporal feature data.

[0132] The sixth spatiotemporal feature data is merged with the third spatiotemporal feature data to obtain the seventh spatiotemporal feature data;

[0133] A feature fusion layer is used to extract and fuse shallow detail features, deep local features and global structural features from the seventh spatiotemporal feature data to obtain the eighth spatiotemporal feature data.

[0134] Multiple pooling layers are used to pool the eighth spatiotemporal feature data to obtain the ninth spatiotemporal feature data;

[0135] A feature fusion layer is used to extract and fuse shallow detail features, deep local features and global structural features from the ninth spatiotemporal feature data to obtain the tenth spatiotemporal feature data.

[0136] The third and second feature extraction layers are used to further extract core features from the tenth spatiotemporal feature data to obtain the eleventh spatiotemporal feature data.

[0137] The eleventh spatiotemporal feature data is input into the inference layer to obtain information on the frequency of head nodding, duration of prone sleeping, and duration of supine sleeping.

[0138] In one possible implementation, the plurality of pooling layers includes one or more of a first max pooling layer, a second max pooling layer, and an average pooling layer.

[0139] In one possible implementation, the processing module 502 is specifically used for:

[0140] A second feature extraction layer and a feature segmentation layer are used to extract shallow detail features from the fourth spatiotemporal feature data and perform feature segmentation to obtain the first shallow detail features and the second shallow detail features.

[0141] A second feature extraction layer and a third feature extraction layer are used to extract deep features from the first shallow layer details to obtain deep local features;

[0142] A third feature extraction layer and a second feature extraction layer are used to extract global structural features from the fourth spatiotemporal feature data;

[0143] A fusion layer is used to fuse the first shallow detail features, the second shallow detail features, the deep local features, and the global structural features to obtain fused feature data;

[0144] A second feature extraction layer is used to extract features from the fused feature data to obtain the fifth spatiotemporal feature data.

[0145] In one possible implementation, the processing module 502 is specifically used for:

[0146] Construct an initial sleep state detection model;

[0147] Obtain training data;

[0148] A total loss function is constructed based on preset head nodding frequency loss function, preset prone sleeping loss function, and preset lying sleeping loss function;

[0149] Based on the total loss function, the initial sleep state detection model is trained using training data to obtain a trained sleep state detection model.

[0150] In one possible implementation, a preset head-nodding frequency loss function is included, comprising:

[0151]

[0152] in, The preset head-nodding frequency loss function is used. For real labels, For predicting labels, This is the first hyperparameter.

[0153] In one possible implementation, a preset prone sleeping loss function is included, comprising:

[0154]

[0155] in, To pre-define the prostrate sleeping loss function, For real labels, For predicting labels, This is the second hyperparameter.

[0156] In one possible implementation, a pre-defined sleep loss function is included, comprising:

[0157]

[0158] in, To preset the sleep loss function, For real labels, For predicting labels, This is the third hyperparameter.

[0159] In one possible implementation, the total loss function includes:

[0160]

[0161] in, For the total loss function, The preset head-nodding frequency loss function is used. To pre-define the prostrate sleeping loss function, To preset the sleep loss function, , and These are the fourth, fifth, and sixth hyperparameters, respectively.

[0162] For a description of the features in the embodiment corresponding to the personnel status detection device, please refer to the relevant description of the embodiment corresponding to the personnel status detection method, which will not be repeated here.

[0163] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0164] In the specific implementation process, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-described personnel status detection method embodiment.

[0165] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0166] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0167] The memory may include high-speed memory (Random Access Memory, RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0168] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0169] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the personnel status detection method when it is run.

[0170] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0171] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the personnel status detection method.

[0172] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the personnel status detection method.

[0173] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] The foregoing has provided a detailed description of the personnel status detection method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for detecting personnel status, characterized in that, include: Obtain video data of personnel; The video data is input into a trained sleep state detection model, which extracts and identifies the spatial position features of the person's head, limbs, and torso, as well as the temporal features of the displacement changes of the head, limbs, and torso from the video data, and outputs the person's head nodding frequency information, prone sleeping duration information, and supine sleeping duration information, including: The first feature extraction layer is used to extract the spatial position features of the person's head, limbs and torso, as well as the temporal features of the displacement changes of the head, limbs and torso, from the video data to obtain the first spatiotemporal feature data. Multiple pooling layers are used to pool the first spatiotemporal feature data to obtain the second spatiotemporal feature data; A second feature extraction layer and a third feature extraction layer are used to further extract core features from the second spatiotemporal feature data to obtain the third spatiotemporal feature data; Multiple pooling layers are used to pool the third spatiotemporal feature data to obtain the fourth spatiotemporal feature data; A feature fusion layer is employed to extract and fuse shallow detail features, deep local features, and global structural features from the fourth spatiotemporal feature data to obtain fifth spatiotemporal feature data, including: The feature fusion layer includes a second feature extraction layer, a third feature extraction layer, a feature segmentation layer, and a fusion layer; Using the second feature extraction layer and the feature segmentation layer, shallow detail features are extracted from the fourth spatiotemporal feature data and feature segmentation is performed to obtain the first shallow detail feature and the second shallow detail feature. The second feature extraction layer and the third feature extraction layer are used to perform deep feature extraction on the first shallow detail features to obtain deep local features; The third feature extraction layer and the second feature extraction layer are used to extract global structural features from the fourth spatiotemporal feature data; The fusion layer is used to fuse the first shallow detail features, the second shallow detail features, the deep local features, and the global structural features to obtain fused feature data. The second feature extraction layer is used to extract features from the fused feature data to obtain the fifth spatiotemporal feature data; An upsampling layer is used to process the feature dimensions of the fifth spatiotemporal feature data to be the same as those of the third spatiotemporal feature data, so as to obtain the sixth spatiotemporal feature data. The sixth spatiotemporal feature data is merged with the third spatiotemporal feature data to obtain the seventh spatiotemporal feature data; A feature fusion layer is used to extract and fuse shallow detail features, deep local features, and global structural features from the seventh spatiotemporal feature data to obtain the eighth spatiotemporal feature data. Multiple pooling layers are used to pool the eighth spatiotemporal feature data to obtain the ninth spatiotemporal feature data; A feature fusion layer is used to extract and fuse shallow detail features, deep local features, and global structural features from the ninth spatiotemporal feature data to obtain the tenth spatiotemporal feature data. The core features of the tenth spatiotemporal feature data are further extracted using the third feature extraction layer and the second feature extraction layer to obtain the eleventh spatiotemporal feature data. The eleventh spatiotemporal feature data is input into the inference layer to obtain the nodding frequency information, prone sleeping duration information, and supine sleeping duration information of the person. If it is determined that the nodding frequency information is greater than a first preset threshold within a preset duration, the prone sleeping duration information is greater than a second preset threshold, and / or the lying-down sleeping duration information is greater than a third preset threshold, then the person is marked as being in a sleep state.

2. The method according to claim 1, characterized in that, The video data of the personnel being acquired includes: The raw video data collected by the personnel; The original video data is preprocessed to obtain preprocessed video data; Obtain the maximum frame rate information that the sleep state detection model can process; If the frame rate information of the preprocessed video data is greater than the maximum processable frame rate information, then the preprocessed video data is subjected to frame discarding processing, so that the frame rate information of the preprocessed video data is less than or equal to the maximum processable frame rate information, so as to obtain the video data of the person. Otherwise, the preprocessed video data will be used as the video data of the person.

3. The method according to claim 2, characterized in that, The preprocessing of the original video to obtain preprocessed video data includes: Each frame of image data in the original video data is processed to a preset size and then filled with grayscale to obtain video data of the preset size; A preset filtering algorithm is used to filter noise in the video data of the preset size to obtain denoised video data; A preset motion blur removal algorithm is used to remove motion blur from the denoised video data to obtain preprocessed video data.

4. The method according to claim 1, characterized in that, Also includes: If the person is marked as asleep, a sleep reminder message is sent to the person's terminal device, and the duration of the person's sleep state is continuously recorded; If the duration of the sleep state exceeds the fourth preset threshold, a sleep warning message is sent to the terminal device of the person's superior management personnel. If the duration of the sleep state exceeds a fifth preset threshold, a sleep alarm message is sent to the terminal device of the person's superior supervisor.

5. The method according to claim 1, characterized in that, The plurality of pooling layers includes one or more of a first max pooling layer, a second max pooling layer, and an average pooling layer.

6. The method according to claim 1, characterized in that, Before inputting the video data into the trained sleep state detection model, the following steps are included: Construct an initial sleep state detection model; Obtain training data; A total loss function is constructed based on preset head nodding frequency loss function, preset prone sleeping loss function, and preset lying sleeping loss function; Based on the total loss function, the initial sleep state detection model is trained using the training data to obtain a trained sleep state detection model.

7. The method according to claim 6, characterized in that, The preset head-nodding frequency loss function includes: in, The preset head-nodding frequency loss function is used. For real labels, For predicting labels, This is the first hyperparameter.

8. The method according to claim 6, characterized in that, The preset prone sleeping loss function includes: in, To pre-define the prostrate sleeping loss function, For real labels, For predicting labels, This is the second hyperparameter.

9. The method according to claim 6, characterized in that, The preset sleep loss function includes: in, To preset the sleep loss function, For real labels, For predicting labels, This is the third hyperparameter.

10. The method according to claim 6, characterized in that, The total loss function includes: in, For the total loss function, The preset head-nodding frequency loss function is used. To pre-define the prostrate sleeping loss function, To preset the sleep loss function, , and These are the fourth, fifth, and sixth hyperparameters, respectively.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the personnel status detection method as described in any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the personnel status detection method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the personnel status detection method as described in any one of claims 1 to 10.

Images