Head posture monitoring method and system based on deep learning

Abstract

The invention discloses a head posture monitoring method and system based on deep learning, and belongs to the field of artificial intelligence. The monitoring method comprises the following steps: constructing a data set for neural network training; training the face head area positioning network and the head posture estimation network; collecting a monitoring image of a target object in real time, and extracting a head region image of the collected image based on a human face head region positioning network; and predicting the head posture of the head area image based on the head posture estimation network, and if the posture reaches a monitoring angle threshold value, carrying out screen switching. According to the method, the monitoring precision of head posture monitoring is effectively improved, and the problem that positioning of a traditional algorithm is not real-time is solved; according to the head posture estimation network, more attention is paid to images near the screen switching angle, the head posture can be estimated in real time, and in the monitoring period, once the preset threshold value is reached, screen switching is conducted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a head posture monitoring method and system based on deep learning. Background Art

[0002] In recent years, with the improvement of computer performance and the rapid development of artificial intelligence technology, the field of computer vision has also developed rapidly, such as face recognition, target detection, posture estimation and other aspects. Computer vision problems based on biometrics have become the research focus of scholars around the world. They not only have extremely high academic value, but also have broad practical application prospects and market potential. The development of head posture detection technology is expected to achieve a smarter and more natural human-computer interaction experience and provide more accurate emotion analysis and behavior recognition capabilities. Head behavior, including facial expressions, body language and visual attention, can reflect our intentions, emotions and attention distribution. Machine learning technology can interpret and understand human behavior non-verbally by analyzing head posture data, providing strong support for human-computer interaction, intelligent monitoring and other fields. For example, nodding behavior usually indicates approval or affirmation, while the driver's head-down behavior may indicate dangerous driving. Using head posture detection technology, these subtle dynamic changes can be accurately captured, thereby improving the intelligence and naturalness of human-computer interaction. Therefore, head posture estimation based on head information can effectively improve the intelligence of human-computer interaction. However, the existing technical routes have defects such as low accuracy and slow processing speed. Therefore, it is necessary to propose a head posture monitoring solution based on deep learning that can effectively improve the processing speed and accuracy. Summary of the invention

[0003] The present invention provides a head posture monitoring method based on deep learning to overcome the defects of the prior art and realize real-time and accurate head posture estimation and monitoring.

[0004] On the one hand, the present invention provides a head posture monitoring method based on deep learning, the method comprising the following steps:

[0005] Step S1, constructing a data set for neural network training;

[0006] Step S2, training the face and head region positioning network and the head posture estimation network respectively based on the data set constructed in step S1 until a preset training condition is met (such as the number of training times reaches a predetermined upper limit or the training loss function value converges), thereby obtaining a trained face and head region positioning network and a trained head posture estimation network;

[0007] When training the head posture estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the posture angle in the sample label of the dataset and the target posture angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0008] Step S3, inputting the real-time collected image to be monitored for head posture, and performing image preprocessing on the image to match the input of the face head area positioning network;

[0009] Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network;

[0010] Step S5, resizing the extracted head region image, and then inputting it into the trained head posture estimation network, obtaining a predicted head posture based on the output of the head posture estimation network; determining whether the currently predicted head posture reaches a preset monitoring angle threshold, and if so, switching the screen;

[0011] Step S6, within the head posture monitoring cycle, repeat steps S3-S5 to achieve continuous monitoring of the monitoring target.

[0012] Furthermore, in step S1, open source image data and user-collected image data are used to construct a data set for training a neural network-based face and head area positioning network; wherein the open source image data include face data sets at different angles and image data containing the head and face (for head area positioning).

[0013] Furthermore, in step S1, the data set constructed in step S1 is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0014] Furthermore, in step S2, the pre-trained YoloV7 algorithm is used as the initial structure of the face head area localization network, and the activation function in the YoloV7 algorithm is set to LeakyReLU to avoid the problem of neuron failure when the neuron input value is less than or equal to 0, thereby optimizing the model performance;

[0015] The face data set in the training set is input into the face head area localization network for network training to obtain the weights of the face head area localization network; the bounding box is corrected through back propagation to gradually reduce the network loss, and the detected head area in the image is predicted. Through the chain derivation rule, the network loss is gradually transferred to the input layer to adjust the network parameters.

[0016] Furthermore, the head posture estimation network is used to predict three angles of the head posture, including: a posture angle prediction value, a range or weight of each discrete value of the posture angle, and an interval between adjacent discrete values ​​of the posture angle.

[0017] Furthermore, the attitude angles include yaw angle, pitch angle and roll angle.

[0018] Furthermore, the head pose estimation network includes: a dual heterogeneous stream structure, a stage fusion module and a soft stage regression network module;

[0019] In the dual heterogeneous stream structure, two stream modules with different structures are included, wherein the first stream module includes in sequence: a first separable convolution block (BR block) with 16 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block includes in sequence: a 2-dimensional depth-separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer), and a ReLU activation function;

[0020] The second stream module includes in sequence: a second separable convolution block (BT block) with 16 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein, the second separable convolution block includes in sequence: a 2-dimensional depth-separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer), and a Tanh activation function;

[0021] In the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivided categories K at each stage;

[0022] Using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element-wise multiplication, and then the fused features are converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, and the output feature map U of the kth stage with a feature dimension of W×H×C is obtained. k , k = 1, 2, ..., K, C is the number of channels of the output feature map, W × H is the output feature map U k width and height; each output feature map U k The corresponding feature space information is stored in the

[0023] The K output feature maps U kThe data is input into the soft stage regression network module (SSR-Net) to predict the three angles of the head posture, and the predicted value of the head posture angle, the range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

[0024] Further, step S4 specifically includes the following steps:

[0025] Determine a bounding box of the head region of the captured image according to a face head region positioning network;

[0026] Determine a bounding box of the head area of ​​the monitored person based on the bounding box of the head area and preset limiting conditions;

[0027] Perform head region detection on the captured image. If the detected head region is not within the head region boundary box, it is determined that there is no person in the region; if the head region cannot be detected, it is determined that there is no person in the region;

[0028] If the detection result shows that there is no one in the area, return to step S3; otherwise, determine the number of heads in the head area boundary box. If there is one, extract the head area image based on the head area in the head area boundary box; if there are multiple heads, perform redundant deletion processing and then extract the head area image based on the area corresponding to the head area boundary box.

[0029] On the other hand, the present invention also provides a head posture monitoring system based on deep learning, which includes: a data acquisition module, an image processing module and a screen switching module;

[0030] The data acquisition module is used to acquire the collected images of the target area to be monitored for head posture in real time, and perform image preprocessing on the images to match the input of the face head area positioning network in the image processing module, and then send the collected images after image preprocessing to the image processing module;

[0031] The image processing module includes a built-in face head region positioning network and a head posture estimation network. The face head region positioning network predicts the head region of the input acquisition image, and extracts the head region image of the acquisition image based on the head positioning region output by the face head region positioning network; the extracted head region image is resized and then sent to the head posture estimation network; the head posture estimation network is used to predict the head posture of the input head region image and transmit the current predicted head posture to the screen switching module in real time;

[0032] The screen switching module determines whether the currently predicted head posture reaches the preset monitoring angle threshold, and if so, performs screen switching;

[0033] Among them, when the head posture estimation network is trained or the model is fine-tuned, the loss weight of each sample is set based on the deviation between the pitch angle in the sample label of the corresponding dataset and the target pitch angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

[0034] The technical solution provided by the present invention brings at least the following beneficial effects:

[0035] The present invention effectively improves the monitoring accuracy of head posture monitoring. Furthermore, the face head area positioning network uses the improved YOLOv7 algorithm to select the head area appearing in the image, solving the problem that the traditional algorithm positioning is not real-time; and the head posture estimation network uses the improved FSANet network, improves the weight part of the sample, and assigns different weight information to different samples. During model training, the contribution values ​​of different samples to the training loss are different, so that the network can pay more attention to the samples near the switching angle, thereby improving the switching accuracy. In order to minimize the loss of the head posture estimation network, the model of the present invention will pay more attention to the part with larger weight, that is, these images near the screen switching angle, and then extract multiple features from these images and fuse them, so that the head posture can be estimated in real time. During the monitoring period, once a preset threshold is reached, the screen will be switched. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a device schematic diagram of a head posture monitoring system based on deep learning;

[0038] Figure 2 It is a structural diagram of a head posture monitoring system based on deep learning;

[0039] Figure 3 A schematic diagram of the processing process of a head posture monitoring method based on deep learning;

[0040] Figure 4 A schematic diagram of the structure of a head posture estimation network used in an embodiment of the present invention;

[0041] Figure numerals: 1 - camera, 2 - display, 3 - computer host. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0043] join Figure 1 and Figure 2 , a head posture monitoring system based on deep learning disclosed in an embodiment of the present invention includes a data acquisition module, such as a camera 1; an image processing module and a screen switching module deployed on a computer host 3, and a display 2, wherein the camera 1 is deployed on the display 2, and is used to acquire head monitoring images when the user uses the display 2, that is, acquired images to be subjected to head posture monitoring. Among them, the data acquisition module is used to acquire the acquired images to be subjected to head posture monitoring in the target area in real time, and perform image preprocessing on the acquired images to match the input of the face head area positioning network in the image processing module, and then send the acquired images after image preprocessing to the image processing module; the image processing module includes a built-in face head area positioning network (such as a face head area positioning network based on the YOLOv7 algorithm) and a head posture estimation network (such as a head posture estimation network based on the FSANet network), the face head area positioning network performs head area prediction on the input acquired image, and extracts the acquired images based on the head positioning area output by the face head area positioning network. The head area image of the image is extracted; the extracted head area image is resized and then sent to the head posture estimation network; the head posture estimation network is used to predict the head posture of the input head area image and transmit the current predicted head posture to the screen switching module in real time; the screen switching module determines whether the current predicted head posture reaches the preset monitoring angle threshold, and if so, the screen is switched; wherein, during training or model fine-tuning, the head posture estimation network sets the loss weight of each sample based on the deviation between the pitch angle in the sample label of the corresponding data set and the target pitch angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

[0044] In the embodiment of the present invention, a weight of 0.5 or 1 is assigned to each sample according to the original annotation information, and the calculation is as follows: For example, the pitch angles of a batch of samples are [10, 20, 15, 20, 19]. In the existing processing, when training or fine-tuning the head posture estimation network, the contribution value of each sample to the loss function value during model training is usually the same. For example, all the contribution weights are set to 1, and the corresponding total loss is: Loss = 1*loss1+1*loss2+1*loss3+1*loss4+1*loss5, and all samples are indistinguishable; In order to improve the accuracy of screen switching triggered by head posture monitoring, this application sets the loss weight of each sample based on the deviation between the pitch angle in the sample label and the target pitch angle, resets the weight of the sample with a pitch angle near the target pitch angle to 1, and the rest to 0.5. The improved total loss is: Loss = 0.5*loss1+1*loss2+0.5*loss3+1*loss4+1*loss5. Therefore, in order to minimize the loss of the network, the model will pay more attention to the part with larger weights, that is, these images near the screen switching angle. By extracting multiple features from the image and fusing them, the head posture can be estimated in real time. During monitoring, once a preset threshold is reached, the screen will be switched.

[0045] In one embodiment, the present invention also provides a head posture monitoring method based on deep learning, see Figure 3 , the specific steps include:

[0046] Step S1, constructing a data set for training a neural network;

[0047] Step S2, training the face and head region positioning network and the head posture estimation network respectively based on the data set constructed in step S1 until a preset training condition is met (such as the number of training times reaches a predetermined upper limit or the training loss function value converges), thereby obtaining a trained face and head region positioning network and a trained head posture estimation network;

[0048] When training the head posture estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the posture angle in the sample label of the dataset and the target posture angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0049] It should be noted that in step 2, the face head area localization network and the head posture estimation network can also be jointly trained based on the data set.

[0050] Step S3, inputting the real-time collected image to be monitored for head posture, and performing image preprocessing on the image to match the input of the face head area positioning network;

[0051] Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network;

[0052] Step S5, resizing the extracted head area image, and then inputting it into the pre-trained and fine-tuned head posture estimation network, obtaining the predicted head posture based on the output of the head posture estimation network; determining whether the currently predicted head posture reaches a preset monitoring angle threshold, and if so, switching the screen;

[0053] Among them, fine-tuning the pre-trained head posture estimation network is specifically as follows:

[0054] Based on the deviation between the pose angle in the sample label of the fine-tuning dataset and the target pose angle, the loss weight of each fine-tuning sample is set: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5;

[0055] Among them, the attitude angle is preferably a pitch angle.

[0056] Step S6, within the head posture monitoring cycle, repeat steps S3-S5 to achieve continuous monitoring of the monitoring target.

[0057] Preferably, in step S1 of the present embodiment, the collected historical image data is used to generate a data set for training the neural network; the historical image data mainly includes a head area data set and a face data set at different angles.

[0058] In one embodiment, the data sets used in step S1 include 300-w, FPW, ELEN, FW and IBUG, and there are about tens of thousands of samples available, from which the angle information of the head can be obtained; the head area data sets at different angles are particularly blurred and the face cannot be distinguished, so they are removed. The removed data sets are annotated, and the head area in the image is manually selected.

[0059] In one embodiment, the face and head area positioning network adopted in this embodiment is an improved YOLOv7 algorithm, and a pre-trained YoloV7 algorithm is used as the initial structure of the face and head area positioning network, and the activation function in the YoloV7 algorithm is set to LeakyReLU to avoid the problem of neuron failure when the neuron input value is less than or equal to 0, thereby optimizing the model performance; and then the network parameters of the initial structure are trained based on the data set constructed in step S1 to obtain a trained face and head area positioning network.

[0060] In one embodiment, in step S2, the data set constructed in step S1 is used to train a neural network, i.e., a face head region positioning network, to obtain a parameter model of the head region positioning network. At the same time, a parameter model of a head region posture estimation network can also be trained based on the data set. The specific implementation steps of step S2 of this embodiment include:

[0061] S21, 80% of the face dataset and face datasets at different angles are used for training, called the training set, 10% are used for verification, called the verification set, and the remaining 10% are used for testing, called the test set;

[0062] S22, inputting the face data set in the data set of step S21 into the face head area localization network for model training to obtain its weight;

[0063] S23, inputting the face data sets at different angles in the data set of step S21 into the head posture estimation network for training, and obtaining a parameter model, which specifically includes the following sub-steps:

[0064] S231, the training process of the head posture estimation network is to input the face data sets at different angles into the network, obtain the output value, compare it with the label information, obtain the loss value, and then use the back propagation to optimize the neural network parameters. Preferably, in this embodiment, the head posture estimation network is an improved FSANet network;

[0065] See also Figure 4, the head posture estimation network adopted in this embodiment includes: a dual heterogeneous stream structure, a stage fusion module (Fine-grained structure aggregation) and a soft stage regression network module (SSR-Net, i.e., SSRModule); in the dual heterogeneous stream structure, two stream modules with different structures are included, wherein the first stream module (stream one) sequentially includes: a first separable convolution block (BR block) with 16 convolution channels, an average pooling layer (AvgPool) with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block sequentially includes: a 2-dimensional depth-separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer) and a ReLU activation function; the second stream module (stream two) sequentially includes: a second separable convolution block (BT block) with 16 convolution channels, a maximum pooling layer (MaxPool) with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein, the second separable convolution block sequentially includes: a 2-dimensional depth-separable convolution layer (SepConv2D) with a convolution kernel of 3×3, a batch normalization operation layer (BN layer) and a Tanh activation function; in the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivided categories K at each stage; using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element-wise multiplication, and then the fused features are converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, and an output feature map U with a feature dimension of W×H×C at the kth stage is obtained. k , k = 1, 2, ..., K, C is the number of channels of the output feature map, W × H is the output feature map U k width and height; each output feature map U k The corresponding feature space information is stored in the K output feature maps U k The data is input into the soft stage regression network module (SSR-Net) to predict the three angles of the head posture, and the predicted value of the head posture angle, the range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

[0066] That is, in this embodiment, the structure of stream one can be decomposed into a one-layer network consisting of BR+average pooling+BR+BR+average pooling and a two-layer network consisting of BR+BR+average pooling+BR+BR. In dual heterogeneous streams, each stream extracts a feature map in K (the number of subdivided categories) at each stage; using the stage fusion module, the features extracted by the two streams are fused at each stage by element-wise multiplication, instead of waiting until the end to fuse; using the stage fusion module, the features extracted by the two streams are fused at each stage by element-wise multiplication, instead of waiting until the end to fuse; the fused features are converted to the convolution channel c using 1x1 convolution, which is the fine-grained structure mapping module. Then, K output feature maps U of size W×H×C are obtained. k , each cell (U k ) store the corresponding feature space information; then output the feature map U k It is sent to SSR-Net to obtain the parameters {p(k), η(k), Δk}, where p(k) is the network's predicted value of the attitude angle (such as yaw, pitch and roll angle), indicating the estimated result of the current attitude angle k, and η(k) is the discrete value range or weight associated with k, which is used in FSANET to discretize the attitude angle and map the continuous angle to the interval of discrete values. The discretization process can help simplify model learning while avoiding the overly precise demand for continuous value prediction. Δk is the width of each interval in the discretization process, that is, the interval between the discrete values ​​of the attitude angle. For example, if the discretization interval is [-90 degrees, -75 degrees], [-75 degrees, -60 degrees], then Δk = 15 degrees. The advantage of this is that the global spatial information encoding is used instead of the pixel encoding as the input of the feature aggregation module, that is, the spatial correlation between features is taken into account. In fine-grained structure mapping, the features are first flattened into a two-dimensional matrix, which contains the cd (i.e., the combination of channel dimension channel and spatial position dimension spatial) spatial features of all feature maps in all orders (this matrix retains the combined features of all channels and spatial positions). Then, the linear combination of n pixel-level features is used to represent the n representative features extracted from the fine-grained aggregation structure to achieve the purpose of dimensionality reduction. Finally, a fully connected layer is used to implement different functions.

[0067] S233 uses data sets from different angles to train the weights of the head posture estimation network, estimates the head angle information at this moment, and provides data support for subsequent attention screen switching.

[0068] In one embodiment, step S3 of the embodiment of the present invention is used to collect real-time images that need to be monitored for head posture, and preprocess the real-time images to obtain preprocessed images. For example, a camera is used to collect monitoring images of the target area in real time. After the real-time monitoring images are captured and acquired by the camera, the collected real-time monitoring images are preprocessed to obtain preprocessed images so as to match the input of the face head area positioning network. In addition, the preprocessing also includes filtering and denoising the real-time monitoring images.

[0069] In one embodiment, in step S4, the preprocessed image is passed to a face head region positioning network to determine a face boundary box to be monitored in the preprocessed image. The face head region positioning network includes head region positioning and cropping the detected relevant region. The steps specifically include:

[0070] S41, quickly determining a bounding box of the head region according to the parameter model of the face head region localization network trained in S22;

[0071] S42, determining a head region boundary box and restriction conditions according to S41, and determining a head region boundary box of the user;

[0072] S421, defining region A to represent the head selection region of the currently input acquisition image predicted by the face head region positioning network. If region A cannot be detected within the head region boundary box, it is considered that no one is in the region; if region A can be detected but region A is not within the head region boundary box, it is also considered that no one is in the region;

[0073] When no one is detected in the area, return to step S3, otherwise execute step S422;

[0074] S422, if there is only one head frame selection area within the area, it is determined that the area within this range is the predicted head area image, and step S5 is executed. If there are multiple face bounding boxes within the specified bounding box, it is necessary to delete irrelevant people to avoid interference, and step S423 is executed;

[0075] S423, considering that the head posture is monitored when using the display, the head movement amplitude will be relatively small, and the head area bounding box obtained in step S41 is used to crop the head area from the entire image according to the bounding box information, and sent to the head posture estimation network to obtain three angle information, namely, the predicted value of the posture angle, the value range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle.

[0076] In one embodiment, step S5 includes:

[0077] S51, cropping the head region bounding box region obtained in step S4 from the preprocessed image, resizing the head region to ensure that the image size input to the multi-feature fusion convolutional neural network module is a 64×64 head region image, and putting it into the head posture estimation network to determine the head posture;

[0078] S52, feature fusion: extracting features, activating features, and fusion-fusing the input head region image, and outputting a fused head region feature map containing rich features;

[0079] In the process of neural network transmission from shallow to deep layers, the features gradually become more and more abstract. Shallow neural networks mostly extract simple features, such as edge information and corner information. The features extracted by deep layers cannot be distinguished by the naked eye, but they contain the overall information of the image.

[0080] S54, determining three angles of the head area image, and judging whether a preset threshold is reached, and once the preset angle is reached, performing screen switching.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0082] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A head posture monitoring method based on deep learning, characterized in that: The following steps are involved: Step S1, constructing a data set for neural network training; Step S2, training the face head region positioning network and the head posture estimation network separately based on the data set constructed in step S1, or jointly training the face head region positioning network and the head posture estimation network until a preset training condition is reached, thereby obtaining a trained face head region positioning network and a trained head posture estimation network; When training the head posture estimation network, the loss weight of each fine-tuning sample is set based on the deviation between the posture angle in the sample label of the dataset and the target posture angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5; Step S3, inputting the real-time collected image to be monitored for head posture, and performing image preprocessing on the image to match the input of the face head area positioning network; Step S4, inputting the collected image after image preprocessing into the trained face head region positioning network, and extracting the head region image of the collected image based on the head positioning region output by the network; Step S5, resizing the extracted head region image, and then inputting it into the trained head posture estimation network, obtaining a predicted head posture based on the output of the head posture estimation network; determining whether the currently predicted head posture reaches a preset monitoring angle threshold, and if so, switching the screen; Step S6, within the head posture monitoring cycle, repeat steps S3-S5 to achieve continuous monitoring of the monitoring target.

2. The method according to claim 1, characterized in that In step S1, a data set is constructed using open source image data and image data collected by users; wherein the open source image data includes face data sets at different angles and image data including the head and face.

3. The method according to claim 1, characterized in that In step S1, the data set constructed in step S1 is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

4. The method according to claim 1, characterized in that In step S2, the pre-trained YoloV7 algorithm is used as the initial structure of the face head area localization network, and the activation function in the YoloV7 algorithm is set to LeakyReLU.

5. The method according to claim 1, characterized in that The head pose estimation network is used to predict three angles of head pose, including: the pose angle prediction value, the range or weight of each discrete value of the pose angle, and the interval between adjacent discrete values ​​of the pose angle.

6. The method according to claim 5, characterized in that The attitude angles are yaw, pitch and roll.

7. The method according to claim 5, characterized in that The head pose estimation network includes: dual heterogeneous stream structure, stage fusion module and soft stage regression network module; In the dual heterogeneous stream structure, two stream modules with different structures are included, wherein the first stream module includes in sequence: a first separable convolution block with 16 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 32 convolution channels, an average pooling layer with a pooling kernel of 2×2, two BR blocks with 64 convolution channels, an average pooling layer with a pooling kernel of 2×2, and two BR blocks with 128 convolution channels; wherein the first separable convolution block includes in sequence: a 2-dimensional depth-separable convolution layer with a convolution kernel of 3×3, a batch normalization operation layer, and a ReLU activation function; The second stream module includes in sequence: a second separable convolution block with 16 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 32 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, two BT blocks with 64 convolution channels, a maximum pooling layer with a pooling kernel of 2×2, and two BT blocks with 128 convolution channels; wherein the second separable convolution block includes in sequence: a 2-dimensional depth-separable convolution layer with a convolution kernel of 3×3, a batch normalization operation layer, and a Tanh activation function; In the dual heterogeneous streams, each stream module extracts a feature map in the number of subdivided categories K at each stage; Using the stage fusion module, the features extracted by the two stream modules are fused at each stage of feature extraction by element-wise multiplication, and then the fused features are converted to the current convolution channel through a convolution layer with a convolution kernel of 1x1, to obtain an output feature map Uk of the kth stage with a feature dimension of W×H×C, k=1,2,…,K, C is the number of channels of the output feature map, and W×H is the width and height of the output feature map Uk; The K output feature maps Uk are input into the soft stage regression network module to predict the three angles of the head posture, and the predicted value of the head posture angle, the value range or weight of each discrete value of the posture angle, and the interval between adjacent discrete values ​​of the posture angle are obtained.

8. The method according to claim 1, characterized in that Step S4 specifically includes the following steps: Determine a bounding box of the head region of the captured image according to a face head region positioning network; Determine a bounding box of the head area of ​​the monitored person based on the bounding box of the head area and preset limiting conditions; Perform head area detection on the captured image. If the detected head area is not within the head area boundary box, it is determined that there is no person in the area. If the head area cannot be detected, it is determined that there is no one in the area; If the detection result is that there is no person in the area, return to step S3; otherwise, determine the number of heads in the head area boundary box, and if there is one, extract the head area image based on the head area in the head area boundary box; If there are multiple images, after performing redundancy removal processing, the head area image is extracted based on the area corresponding to the head area boundary box.

9. A head posture monitoring system based on deep learning, characterized in that: It includes data acquisition module, image processing module and screen switching module; The data acquisition module is used to acquire the collected images of the target area to be monitored for head posture in real time, and perform image preprocessing on the images to match the input of the face head area positioning network in the image processing module, and then send the collected images after image preprocessing to the image processing module; The image processing module includes a built-in face head region positioning network and a head posture estimation network. The face head region positioning network predicts the head region of the input acquisition image, and extracts the head region image of the acquisition image based on the head positioning region output by the face head region positioning network; the extracted head region image is resized and then sent to the head posture estimation network; the head posture estimation network is used to predict the head posture of the input head region image and transmit the current predicted head posture to the screen switching module in real time; The screen switching module determines whether the currently predicted head posture reaches the preset monitoring angle threshold, and if so, performs screen switching; Among them, when the head posture estimation network is trained or the model is fine-tuned, the loss weight of each sample is set based on the deviation between the posture angle in the sample label of the corresponding dataset and the target posture angle: if the deviation is greater than or equal to the specified threshold, the weight is set to 1, otherwise it is set to 0.5.

Images