A head posture monitoring method and system based on deep learning

Abstract

The present invention discloses a head posture monitoring method and system based on deep learning, which belongs to the field of artificial intelligence. The monitoring method of the present invention includes: constructing a data set for neural network training; training a face head region positioning network and a head posture estimation network; real-time acquisition of monitoring images of the target object, extracting a head region image of the acquired image based on the face head region positioning network; and then predicting the head posture of the head region image based on the head posture estimation network. If the posture reaches the monitoring angle threshold, the screen is switched. The present invention effectively improves the monitoring accuracy of head posture monitoring and solves the problem that traditional algorithms are not positioned in real time; the head posture estimation network of the present invention pays more attention to images near the screen switching angle, and can estimate the head posture in real time. During the monitoring period, once a preset threshold is reached, the screen will be switched.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a head posture monitoring method and system based on deep learning. Background Art

[0002] In recent years, with the advancement of computer performance and the rapid development of artificial intelligence technology, the field of computer vision has also experienced rapid growth, encompassing areas such as face recognition, object detection, and pose estimation. Biometric-based computer vision problems have become a research focus for scholars worldwide. These problems not only possess high academic value but also offer broad practical application prospects and market potential. The development of head pose detection technology promises to enable smarter and more natural human-computer interaction experiences, providing more accurate emotion analysis and behavior recognition capabilities. Head behavior, including facial expressions, body language, and visual attention, can reflect our intentions, emotions, and attention distribution. Machine learning techniques can analyze head pose data to interpret and understand human behavior non-verbally, providing powerful support for areas such as human-computer interaction and intelligent monitoring. For example, nodding often indicates approval or affirmation, while a driver's head down may indicate dangerous driving. Head pose detection technology can accurately capture these subtle dynamic changes, thereby enhancing the intelligence and naturalness of human-computer interaction. Therefore, head pose estimation based on head information can effectively enhance the intelligence of human-computer interaction. However, the existing technical routes have defects such as low accuracy and slow processing speed. Therefore, it is necessary to propose a head posture monitoring solution based on deep learning that can effectively improve processing speed and accuracy. Summary of the Invention

[0003] The present invention provides a head posture monitoring method based on deep learning to overcome the shortcomings of the existing technology and achieve real-time and accurate head posture estimation and monitoring.

[0004] In one aspect, the present invention provides a head posture monitoring method based on deep learning, the method comprising the following steps:

[0005] Step S1, constructing a data set for neural network training;

[0006] Step S2: training the face and head region localization network and the head pose estimation network based on the data set constructed in step S1 until a preset training condition is met (e.g., the number of training times reaches a predetermined upper limit or the training loss function value converges), thereby obtaining a trained face and head region localization network and a trained head pose estimation network;

[0007] When training the head pose estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the pose angle in the sample label of the dataset and the target pose angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0008] Step S3: inputting the real-time collected image to be used for head posture monitoring, and performing image preprocessing on the image to match the input of the face head region localization network;

[0009] Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network;

[0010] Step S5: resize the extracted head region image and input it into the trained head pose estimation network. Based on the output of the head pose estimation network, a predicted head pose is obtained. It is determined whether the currently predicted head pose reaches a preset monitoring angle threshold. If so, the screen is switched.

[0011] Step S6: within the head posture monitoring period, repeat steps S3-S5 to achieve continuous monitoring of the monitoring target.

[0012] Furthermore, in step S1, open source image data and user-collected image data are used to construct a dataset for training a neural network-based face and head area positioning network; wherein, the open source image data includes face datasets at different angles and image data containing the head and face (for head area positioning).

[0013] Furthermore, in step S1, the dataset constructed in step S1 is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0014] Furthermore, in step S2, the pre-trained YoloV7 algorithm is used as the initial structure of the face head area localization network, and the activation function in the YoloV7 algorithm is set to LeakyReLU to avoid the problem of neuron failure when the neuron input value is less than or equal to 0, thereby optimizing the model performance;

[0015] The face dataset in the training set is input into the face head area localization network for network training to obtain the weight of the face head area localization network; the bounding box is corrected through back propagation to gradually reduce the network loss, and the detected head area in the image is predicted. Through the chain derivation rule, the network loss is gradually transferred to the input layer to adjust the network parameters.

[0016] Furthermore, the head pose estimation network is used to predict three angles of head pose, including: a predicted pose angle value, a range or weight of each discrete value of the pose angle, and an interval between adjacent discrete values ​​of the pose angle.

[0017] Furthermore, the attitude angles include yaw angle, pitch angle and roll angle.

[0018] Furthermore, the head pose estimation network includes: a dual heterogeneous stream structure, a stage fusion module and a soft stage regression network module;

[0019] In the dual heterogeneous stream structure, two stream modules with different structures are included, wherein the first stream module sequentially includes: a first separable convolution block (BR block) with 16 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block sequentially includes: a 2D depthwise separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer), and a ReLU activation function;

[0020] The second stream module includes, in sequence: a second separable convolution block (BT block) with 16 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein, the second separable convolution block includes, in sequence: a 2D depthwise separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer), and a Tanh activation function;

[0021] In the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivided categories K at each stage;

[0022] Using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element multiplication, and then the fused features are converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, and the output feature map U of the kth stage with a feature dimension of W×H×C is obtained. k , k=1,2,…,K, C is the number of channels of the output feature map, W×H is the output feature map U k width and height; each output feature map U k The corresponding feature space information is stored in

[0023] The K output feature maps U kThe three angles of the head posture are input into the soft stage regression network module (SSR-Net) to predict the posture angle of the head, and the predicted value of the posture angle, the range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

[0024] Furthermore, step S4 specifically includes the following steps:

[0025] Determine the bounding box of the head area of ​​the captured image based on the face head area positioning network;

[0026] Determine the bounding box of the head area of ​​the monitored person based on the bounding box of the head area and the preset restriction conditions;

[0027] Perform head region detection on the captured image. If the detected head region is not within the head region bounding box, it is determined that there is no person in the region. If the head region cannot be detected, it is determined that there is no person in the region.

[0028] If the detection result shows that there is no person in the area, return to step S3; otherwise, determine the number of heads in the head area bounding box. If there is only one head, extract the head area image based on the head area in the head area bounding box; if there are multiple heads, perform redundant deletion processing and then extract the head area image based on the area corresponding to the head area bounding box.

[0029] On the other hand, the present invention also provides a head posture monitoring system based on deep learning, which includes: a data acquisition module, an image processing module and a screen switching module;

[0030] The data acquisition module is used to collect the collected images of the target area to be monitored for head posture in real time, and perform image preprocessing on them to match the input of the face head area positioning network in the image processing module, and then send the collected images after image preprocessing to the image processing module;

[0031] The image processing module includes a built-in face head region localization network and a head pose estimation network. The face head region localization network predicts the head region of the input captured image and extracts the head region image of the captured image based on the head localization region output by the face head region localization network. The extracted head region image is resized and then fed into the head pose estimation network. The head pose estimation network is used to predict the head pose of the input head region image and transmit the current predicted head pose to the screen switching module in real time.

[0032] The screen switching module determines whether the currently predicted head posture reaches the preset monitoring angle threshold. If so, the screen is switched;

[0033] Among them, when the head pose estimation network is trained or fine-tuned, the loss weight of each sample is set based on the deviation between the pitch angle in the sample label of the corresponding dataset and the target pitch angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

[0034] The technical solution provided by the present invention brings at least the following beneficial effects:

[0035] The present invention effectively improves the monitoring accuracy of head posture monitoring. Furthermore, the face head area positioning network uses the improved YOLOv7 algorithm to select the head area appearing in the image, solving the problem that the traditional algorithm positioning is not real-time; and the head posture estimation network uses the improved FSANet network, which improves the weight part of the sample and assigns different weight information to different samples. During model training, the contribution values ​​of different samples to the training loss are different, so that the network can pay more attention to the samples near the switching angle, thereby improving the accuracy of the switching. In order to minimize the loss of the head posture estimation network, the model of the present invention will pay more attention to the parts with larger weights, that is, these images near the screen switching angle, and then extract multiple features from these images and fuse them, so that the head posture can be estimated in real time. During the monitoring period, once a preset threshold is reached, the screen will be switched. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a schematic diagram of a head posture monitoring system based on deep learning;

[0038] Figure 2 This is a structural diagram of a head posture monitoring system based on deep learning;

[0039] Figure 3 Schematic diagram of the processing process of a head posture monitoring method based on deep learning;

[0040] Figure 4 This is a schematic diagram of the structure of the head posture estimation network used in an embodiment of the present invention;

[0041] Figure numerals: 1 - camera, 2 - display, 3 - computer host. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0043] join Figure 1 and Figure 2 , a head posture monitoring system based on deep learning disclosed in an embodiment of the present invention includes a data acquisition module, such as a camera 1; an image processing module and a screen switching module deployed on a computer host 3, and a display 2. The camera 1 is deployed on the display 2 and is used to acquire head monitoring images when the user uses the display 2, that is, the acquired images to be subjected to head posture monitoring. Among them, the data acquisition module is used to acquire the acquired images to be subjected to head posture monitoring in the target area in real time, and perform image preprocessing on the acquired images to match the input of the face head area positioning network in the image processing module, and then send the acquired images after image preprocessing to the image processing module; the image processing module includes a built-in face head area positioning network (such as a face head area positioning network based on the YOLOv7 algorithm) and a head posture estimation network (such as a head posture estimation network based on the FSANet network), the face head area positioning network performs head area prediction on the input acquired image, and extracts the acquired images based on the head positioning area output by the face head area positioning network. The head area image of the image is extracted; the extracted head area image is resized and then sent to the head posture estimation network; the head posture estimation network is used to predict the head posture of the input head area image and transmit the current predicted head posture to the screen switching module in real time; the screen switching module determines whether the current predicted head posture reaches the preset monitoring angle threshold, and if so, performs screen switching; wherein, during training or model fine-tuning, the head posture estimation network sets the loss weight of each sample based on the deviation between the pitch angle in the sample label of the corresponding data set and the target pitch angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

[0044] In the embodiment of the present invention, a weight of 0.5 or 1 is assigned to each sample according to the original annotation information, and the calculation is as follows: For example, the pitch angles of a batch of samples are [10, 20, 15, 20, 19]. In the existing processing, when training or fine-tuning the head posture estimation network, the contribution value of each sample to the loss function value during model training is usually the same. For example, if all the contribution weights are set to 1, the corresponding total loss is: Loss = 1*loss1+1*loss2+1*loss3+1*loss4+1*loss5, and all samples are the same; In order to improve the accuracy of screen switching triggered by head posture monitoring, this application sets the loss weight of each sample based on the deviation between the pitch angle in the sample label and the target pitch angle, resets the weight of samples with pitch angles near the target pitch angle to 1, and the rest are set to 0.5. The improved total loss is: Loss = 0.5*loss1+1*loss2+0.5*loss3+1*loss4+1*loss5. Therefore, in order to minimize the loss of the network, the model will pay more attention to the parts with larger weights, that is, these images near the screen switching angle. Multiple features are extracted from the image and fused to estimate the head posture in real time. During the monitoring period, once a preset threshold is reached, the screen will be switched.

[0045] In one embodiment, the present invention also provides a head posture monitoring method based on deep learning, see Figure 3 , the specific steps include:

[0046] Step S1, constructing a data set for training a neural network;

[0047] Step S2: training the face and head region localization network and the head pose estimation network based on the data set constructed in step S1 until a preset training condition is met (e.g., the number of training times reaches a predetermined upper limit or the training loss function value converges), thereby obtaining a trained face and head region localization network and a trained head pose estimation network;

[0048] When training the head pose estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the pose angle in the sample label of the dataset and the target pose angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0049] It should be noted that in step 2, the face head area positioning network and the head posture estimation network can also be jointly trained based on the data set.

[0050] Step S3: inputting the real-time collected image to be used for head posture monitoring, and performing image preprocessing on the image to match the input of the face head region localization network;

[0051] Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network;

[0052] Step S5: resize the extracted head region image and then input it into a pre-trained and fine-tuned head pose estimation network. A predicted head pose is obtained based on the output of the head pose estimation network. A determination is made as to whether the currently predicted head pose reaches a preset monitoring angle threshold. If so, a screen switch is performed.

[0053] Among them, fine-tuning the pre-trained head pose estimation network is specifically as follows:

[0054] Based on the deviation between the pose angle in the sample label of the fine-tuning dataset and the target pose angle, the loss weight of each fine-tuning sample is set: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0055] Among them, the attitude angle is preferably the pitch angle.

[0056] Step S6: within the head posture monitoring period, repeat steps S3-S5 to achieve continuous monitoring of the monitoring target.

[0057] Preferably, in step S1 of this embodiment, the collected historical image data is used to generate a data set for training the neural network; the historical image data mainly includes a head area data set and a face data set at different angles.

[0058] In one embodiment, the datasets used in step S1 include 300-w, FPW, ELEN, FW, and IBUG, with approximately tens of thousands of samples available, from which head angle information can be obtained. Head regions at different angles that are particularly blurry and unable to distinguish faces are removed from the dataset. The removed datasets are annotated, and the head region in the image is manually selected.

[0059] In one embodiment, the face and head area positioning network used in this embodiment is an improved YOLOv7 algorithm. The pre-trained YoloV7 algorithm is used as the initial structure of the face and head area positioning network, and the activation function in the YoloV7 algorithm is set to LeakyReLU to avoid the problem of neuron failure when the neuron input value is less than or equal to 0, thereby optimizing the model performance; the network parameters of the initial structure are then trained based on the data set constructed in step S1 to obtain a trained face and head area positioning network.

[0060] In one embodiment, step S2 uses the dataset constructed in step S1 to train a neural network, namely, a face head region localization network, to obtain a parameter model of the head region localization network. Simultaneously, a parameter model of a head region posture estimation network can also be trained based on the dataset. The specific implementation steps of step S2 in this embodiment include:

[0061] S21, 80% of the face dataset and the face dataset at different angles are used for training, called the training set, 10% are used for verification, called the validation set, and the remaining 10% are used for testing, called the test set;

[0062] S22, inputting the face data set in the data set of step S21 into the face head region localization network for model training to obtain its weight;

[0063] S23, inputting the face data sets of different angles in the data set of step S21 into the head posture estimation network for training, and obtaining a parameter model, which specifically includes the following sub-steps:

[0064] S231, the training process of the head pose estimation network is to input face datasets at different angles into the network, obtain output values, compare them with the labeled information, obtain loss values, and then use backpropagation to optimize the neural network parameters. Preferably, in this embodiment, the head pose estimation network is an improved FSANet network;

[0065] See also Figure 4The head pose estimation network used in this embodiment includes: a dual heterogeneous stream structure, a stage fusion module (Fine-grained structure aggregation) and a soft stage regression network module (SSR-Net, i.e., SSRModule); in the dual heterogeneous stream structure, it includes two stream modules with different structures, wherein the first stream module (stream one) sequentially includes: a first separable convolution block (BR block) with 16 convolution channels, an average pooling layer (AvgPool) with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block sequentially includes: a 2-dimensional depth-wise separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer), and a ReLU activation function; the second stream module (stream two) sequentially includes: a second separable convolution block (BT block) with 16 convolution channels, a maximum pooling layer (MaxPool) with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein, the second separable convolution block sequentially includes: a 2-dimensional depth-wise separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer) and a Tanh activation function; in the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivision categories K at each stage; using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element-wise multiplication, and the fused features are then converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, obtaining an output feature map U with a feature dimension of W×H×C at the kth stage. k , k=1,2,…,K, C is the number of channels of the output feature map, W×H is the output feature map U k width and height; each output feature map U k The corresponding feature space information is stored in the K output feature maps U k The three angles of the head posture are input into the soft stage regression network module (SSR-Net) to predict the posture angle of the head, and the predicted value of the posture angle, the range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

[0066] That is, in this embodiment, the structure of stream one can be decomposed into a one-layer network consisting of BR+average pooling+BR+BR+average pooling and a two-layer network consisting of BR+BR+average pooling+BR+BR. In dual heterogeneous streams, each stream extracts a feature map in K (the number of subdivided categories) at each stage; using the stage fusion module, the features extracted by the two streams are fused at each stage by element-wise multiplication, instead of waiting until the end to fuse; using the stage fusion module, the features extracted by the two streams are fused at each stage by element-wise multiplication, instead of waiting until the end to fuse; the fused features are converted to the convolution channel c using 1x1 convolution, which is the fine-grained structure mapping module. Then, K output feature maps U of size W×H×C are obtained. k , each cell (U k ) store the corresponding feature space information; then output feature map U k This is fed into the SSR-Net to obtain the parameters {p(k), η(k), Δk}. Here, p(k) is the network's predicted value for attitude angles (such as yaw, pitch, and roll), representing the current estimate of attitude angle k. η(k) is the discrete value range or weight associated with k. In FSANET, this is used to discretize attitude angles, mapping continuous angles to discrete value intervals. Discretization can help simplify model learning while avoiding the need for overly precise continuous value predictions. Δk is the width of each discretization interval, or the interval between discrete attitude angle values. For example, if the discretization intervals are [-90 degrees, -75 degrees], [-75 degrees, -60 degrees], then Δk = 15 degrees. This approach has the advantage of using global spatial information encoding instead of pixel encoding as the input to the feature aggregation module, thus taking into account spatial correlation between features. In fine-grained structure mapping, the features are first flattened into a two-dimensional matrix. The matrix contains the cd (i.e., the combination of the channel dimension and the spatial dimension) spatial features of all feature maps at all orders (this matrix retains the combined features of all channels and spatial positions). Then, the n representative features extracted from the fine-grained aggregate structure are represented using a linear combination of n pixel-level features to achieve the purpose of dimensionality reduction. Finally, a fully connected layer is used to implement different functions.

[0067] S233 uses data sets from different angles to train the weights of the head posture estimation network, estimates the head angle information at this moment, and provides data support for subsequent attention screen switching.

[0068] In one embodiment, step S3 of the present invention is used to capture a real-time image for head posture monitoring and preprocess the real-time image to obtain a preprocessed image. For example, a camera is used to capture a monitoring image of a target area in real time. After the real-time monitoring image is captured and acquired by the camera, the captured real-time monitoring image is preprocessed to obtain a preprocessed image so that it matches the input of the face head region localization network. In addition, the preprocessing also includes filtering and noise reduction of the real-time monitoring image.

[0069] In one embodiment, in step S4, the pre-processed image is fed into a face head region localization network to determine a face bounding box to be monitored in the pre-processed image. The face head region localization network includes head region localization and cropping of the detected relevant region. The steps specifically include:

[0070] S41, quickly determining a bounding box of the head region based on the parameter model of the face head region localization network trained in S22;

[0071] S42, determining the head region bounding box and the restriction conditions according to S41, and determining the head region bounding box of the user;

[0072] S421, defining region A as the head region of the current input captured image predicted by the face head region localization network. If region A cannot be detected within the head region bounding box, it is considered that no one is in the region. If region A can be detected but is not within the head region bounding box, it is also considered that no one is in the region.

[0073] If no one is detected in the area, the process returns to step S3, otherwise the process goes to step S422;

[0074] S422: If only one head bounding box is within the region, the region is determined to be the predicted head region image, and step S5 is executed. If multiple face bounding boxes are within the specified bounding box, irrelevant people need to be deleted to avoid interference, and step S423 is executed.

[0075] S423: Considering that the head posture when using the display is monitored, the head movement amplitude will be relatively small. The head area bounding box obtained in step S41 is used to crop the head area from the entire image according to the bounding box information. The head area is then sent to the head posture estimation network to obtain three angle information, namely, the predicted value of the posture angle, the value range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle.

[0076] In one embodiment, step S5 includes:

[0077] S51, cropping the head region bounding box obtained in step S4 from the preprocessed image, resizing the head region to ensure that the image size input to the multi-feature fusion convolutional neural network module is a 64×64 head region image, and placing it into the head pose estimation network to determine the head pose;

[0078] S52, feature fusion: the input head region image is subjected to feature extraction, feature activation and feature fusion, and a fused head region feature map containing rich features is output;

[0079] As the neural network progresses from shallow layers to deep layers, features become increasingly abstract. Shallow layers of the neural network extract mostly simple features, such as edge information and corner information. Deep layers extract features that cannot be distinguished by the naked eye, but contain information about the entire image.

[0080] S54, determining three angles of the head region image and judging whether a preset threshold is reached. Once the preset threshold is reached, screen switching is performed.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0082] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A head posture monitoring method based on deep learning, characterized in that: The following steps are involved: Step S1, constructing a data set for neural network training; Step S2: training the face and head region localization network and the head pose estimation network separately based on the data set constructed in step S1, or jointly training the face and head region localization network and the head pose estimation network until preset training conditions are met, thereby obtaining trained face and head region localization network and head pose estimation network; When training the head pose estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the pose angle in the sample label of the dataset and the target pose angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5; Step S3: inputting the real-time collected image to be used for head posture monitoring, and performing image preprocessing on the image to match the input of the face head region localization network; Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network; Step S5: resize the extracted head region image and input it into the trained head pose estimation network. Based on the output of the head pose estimation network, a predicted head pose is obtained. It is determined whether the currently predicted head pose reaches a preset monitoring angle threshold. If so, the screen is switched. Step S6: Repeat steps S3-S5 within the head posture monitoring period to achieve continuous monitoring of the monitoring target.

2. The method according to claim 1, wherein In step S1, a dataset is constructed using open source image data and user-collected image data; wherein the open source image data includes face datasets from different angles and image data containing the head and face.

3. The method according to claim 1, wherein In step S1, the dataset constructed in step S1 is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

4. The method according to claim 1, wherein In step S2, the pre-trained YoloV7 algorithm is used as the initial structure of the face head area positioning network, and the activation function in the YoloV7 algorithm is set to LeakyReLU.

5. The method according to claim 1, wherein The head pose estimation network is used to predict three angles of head pose, including: the pose angle prediction value, the range or weight of each discrete value of the pose angle, and the interval between adjacent discrete values ​​of the pose angle.

6. The method according to claim 5, wherein The attitude angles are yaw, pitch and roll.

7. The method according to claim 5, wherein The head pose estimation network includes: dual heterogeneous stream structure, stage fusion module and soft stage regression network module; In the dual heterogeneous stream structure, two stream modules with different structures are included, wherein the first stream module sequentially includes: a first separable convolution block with 16 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block sequentially includes: a 2-dimensional depthwise separable convolution layer with a convolution kernel of 3×3, a batch normalization operation layer, and a ReLU activation function; The second stream module includes, in sequence: a second separable convolution block with 16 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein, the second separable convolution block includes, in sequence: a 2-dimensional depthwise separable convolution layer with a convolution kernel of 3×3, a batch normalization operation layer, and a Tanh activation function; In the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivided categories K at each stage; Using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element-wise multiplication. The fused features are then converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, and the output feature map Uk of the kth stage is obtained with a feature dimension of W×H×C, where k=1,2,…,K, C is the number of channels of the output feature map, and W×H is the width and height of the output feature map Uk; The K output feature maps Uk are input into the soft stage regression network module to predict the three angles of the head posture, and the predicted value of the head posture angle, the value range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

8. The method according to claim 1, wherein Step S4 specifically includes the following steps: Determine the bounding box of the head area of ​​the captured image based on the face head area positioning network; Determine the bounding box of the head area of ​​the monitored person based on the bounding box of the head area and the preset restriction conditions; Perform head area detection on the captured image. If the detected head area is not within the head area boundary box, it is determined that there is no person in the area. If the head area cannot be detected, it is determined that there is no one in the area; If the detection result shows that there is no person in the area, return to step S3; otherwise, determine the number of heads within the head area bounding box. If there is only one head, extract the head area image based on the head area within the head area bounding box; If there are multiple images, after performing redundancy removal processing, the head region image is extracted based on the region corresponding to the head region boundary box.

9. A head posture monitoring system based on deep learning, characterized in that: Including data acquisition module, image processing module and screen switching module; The data acquisition module is used to collect the collected images of the target area to be monitored for head posture in real time, and perform image preprocessing on them to match the input of the face head area positioning network in the image processing module, and then send the collected images after image preprocessing to the image processing module; The image processing module includes a built-in face head region localization network and a head pose estimation network. The face head region localization network predicts the head region of the input captured image and extracts the head region image of the captured image based on the head localization region output by the face head region localization network. The extracted head region image is resized and then fed into the head pose estimation network. The head pose estimation network is used to predict the head pose of the input head region image and transmit the current predicted head pose to the screen switching module in real time. The screen switching module determines whether the currently predicted head posture reaches the preset monitoring angle threshold. If so, the screen is switched; Among them, when the head pose estimation network is trained or fine-tuned, the loss weight of each sample is set based on the deviation between the pose angle in the sample label of the corresponding dataset and the target pose angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

Images