A head posture positioning and detection method and its application and system

Abstract

The present invention discloses a head posture positioning and detection method and its application and system. The head posture positioning and detection method comprises the following steps: obtaining an image set of the head posture to be measured, and performing standard brightness conversion on all images in the image set; using a person detection model and a head detection model to obtain full body position information Pa and head position information Ph for all converted images respectively, inputting the full body position information Pa and the head position information Ph into a pre-built deep learning convolutional neural network, and converting to obtain intermediate features Pa^ and Ph^ of the two position information; fusing the intermediate features Pa^ and Ph^ of the two position information according to a certain weight, and calculating the head deflection Euler angle according to the fused feature P_site. The head posture positioning and detection method provided by the present invention is less affected and has accurate detection; at the same time, it can also effectively improve the rate of model learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of posture detection, and more specifically, relates to a head posture positioning and detection method and an application and system thereof. Background Art

[0002] Head posture detection technology has a wide range of applications, such as driver fatigue detection, classroom attention detection, collaborative expression recognition, etc. On the one hand, although the depth information of the image obtained by the depth camera can make the head posture estimation very accurate, in real life, the cost of depth cameras is high and not universal. On the other hand, the traditional head posture estimation method can show good performance for high-quality images or videos, but the head posture estimation effect is poor for low-resolution images or people in distant scenes, and it is difficult to apply in practice.

[0003] In real environments, light intensity changes, which can interfere with head pose estimation. When a person in an image or video has a large facial occlusion or a large head deflection, traditional machine learning-based head pose estimation methods have difficulty detecting the person's head, and therefore cannot perform head pose estimation based on this.

[0004] Deep learning methods have better performance in the field of images and are more suitable for head posture estimation in real scenes than machine learning methods. This is reflected in the following three aspects: (1) they show good robustness in low-resolution scenes; (2) they are insensitive to changes in light in the scene; and (3) they can repair occluded and deflected portraits in the image through certain methods (such as generative adversarial networks), thereby improving the prediction accuracy.

[0005] At the same time, deep learning methods also have certain defects when performing head posture estimation tasks: in terms of detection effect, the estimation effect of head posture is directly affected by the quality of the model; in terms of time cost, deep learning methods often take a lot of time to train the model, and during the model training process, irrelevant background factors will interfere with the training and slow down the overall optimization rate of parameters. Summary of the invention

[0006] In view of the defects of the prior art, the purpose of the present invention is to provide a head posture positioning and detection method and its application and system, aiming to solve the problems existing in traditional head posture estimation (susceptible to objective factors such as ambient light, and subjective factors such as head deflection and occlusion), while improving the rate of model learning.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a head posture positioning and detection method, comprising the following steps:

[0008] (1) obtaining an image set of the head posture to be measured, and performing standard brightness conversion on all images in the image set, wherein the image set includes an image set arranged in sequence in image data, or an image set arranged in frame column sequence in video data;

[0009] (2) using a person detection model and a head detection model to obtain full body position information Pa and head position information Ph for all images after standard brightness conversion, inputting the full body position information Pa and the head position information Ph into a pre-built deep learning convolutional neural network, and converting them to obtain intermediate features Pa^ and Ph^ of the two position information;

[0010] (3) The intermediate features Pa^ and Ph^ of the two types of position information are fused according to a certain weight, and the head deflection Euler angle is calculated based on the fused feature P_site.

[0011] In one embodiment, step (1) specifically includes:

[0012] For image data, the R, G, and B channels of each image in the image set are directly converted to a Y single-channel image with standard brightness. For video data, three consecutive frames of images in the image set are obtained, and the R, G, and B channels of each image in the three frames are converted to standard brightness to obtain three frames of brightness information associated with short-term frames (Y1, Y2, Y3), wherein the brightness information Y2 of the middle frame image in the three frames is the reference brightness, and the brightness information Y1 and Y3 of the previous and next two frames are the transformed brightness.

[0013] In one embodiment, step (1) further includes:

[0014] The acquired image set is preprocessed, and all images in the preprocessed image set are subjected to standard brightness conversion, wherein the preprocessing includes one or more of rotation, mirroring, correction alignment and decentering cropping.

[0015] In one of the embodiments, the standard brightness conversion formula is: Y=sqrt(0.241*R^2+0.691*G^2+0.068*B^2).

[0016] In one embodiment, the intermediate features Pa^ and Ph^ are fused as follows:

[0017] P_site=(0.4Pa^⊙E+(0.6Ph^⊙E+α))

[0018] Among them, ⊙ represents the dot product between elements; α is a 0 constant matrix; Pa^⊙E represents the unification of the intermediate features obtained after the convolution layer operation of the deep learning convolutional neural network on the whole body position information Pa and the input dimension of the fully connected layer of the deep learning convolutional neural network; Ph^⊙E represents the unification of the intermediate features obtained after the convolution layer operation of the deep learning convolutional neural network on the head position information Ph and the input dimension of the fully connected layer of the deep learning convolutional neural network.

[0019] In one embodiment, the workflow of the deep learning convolutional neural network is:

[0020] After one convolution layer conv1, one normalization layer bn1, linear unit correction ReLU, maximum pooling operation Max-pooling, and then four layers of upsampling operations, the results are average-pooled mean-pooling and then sent to the fully connected layer.

[0021] In one embodiment, the deep learning convolutional neural network is trained using the UPAN dataset.

[0022] In one of the embodiments, the loss function of the deep learning convolutional neural network adopts mean square error loss.

[0023] In a second aspect, the present invention provides an application of the above-mentioned head posture positioning and detection method in driving warning, classroom attention detection, and children's behavior induction.

[0024] In a third aspect, the present invention provides a head posture positioning and detection system, comprising:

[0025] An image set input module, used to obtain an image set of the head posture to be measured, and perform standard brightness conversion on all images in the image set, wherein the image set includes an image set arranged in sequence in the image data, or an image set arranged in frame column sequence in the video data;

[0026] A position information fusion module is used to obtain full body position information Pa and head position information Ph from all images after standard brightness conversion using a person detection model and a head detection model, input the full body position information Pa and the head position information Ph into a pre-built deep learning convolutional neural network, and convert them to obtain intermediate features Pa^ and Ph^ of the two types of position information;

[0027] The deep learning module is used to fuse the intermediate features Pa^ and Ph^ of the two types of position information according to a certain weight, and calculate the head deflection Euler angle based on the fused feature P_site.

[0028] The head posture positioning and detection method and its application and system provided by the present invention have the following beneficial effects: 1) performing standard brightness conversion on the acquired image set can remove color information irrelevant to the head posture estimation, and at the same time, the brightness dimension Y is obtained by a linear combination of R, G, and B, thereby retaining the information of the original dimension; 2) different models are used for the person detection model and the head detection model, thereby improving the overall robustness of the model; the information obtained by the two models is output through the same network, thereby ensuring that the intermediate features of the two positions have the same spatial structure; 3) the intermediate features of the two positions are fused, and the fused position features retain the accuracy of the head position information and the fault tolerance of the whole body position information, thereby improving the learning rate of the model and reducing the angular deviation of the head posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of a head posture positioning and detection method in one embodiment;

[0030] Figure 2 Schematic diagram of modules of a head posture positioning and detection system in one embodiment;

[0031] Figure 3 is a technical block diagram of a head posture positioning and detection system in one embodiment;

[0032] Figure 4 is a diagram showing the calculation principle of the standard brightness of an image in one embodiment;

[0033] Figure 5 for Figure 4 Flowchart for calculating the standard brightness of the image;

[0034] Figure 6 is a schematic diagram of the principle of fusing head position information and whole body position information in one embodiment;

[0035] Figure 7 Schematic diagram of head posture estimation in a classroom scene in one embodiment;

[0036] Figure 8 is a curve diagram of the deflection angle of the student's head in one embodiment;

[0037] Fig. 9 This is a figure position information diagram obtained by background difference in one embodiment;

[0038] Fig.10 This is a diagram showing the effect of detecting a person's position in a real scene in an embodiment;

[0039] Fig.11 A comparison chart of the face detection model (right) and the dlib model (left) provided by the present invention in real-world scenarios;

[0040] Fig.121. A diagram showing dynamic head border adjustment (a) and model attention convergence (b) in one embodiment;

[0041] Fig.13 A line graph of data set loss in one embodiment;

[0042] Fig.14 Schematic diagram of an ablation experiment in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] In order to solve the problems existing in traditional head posture estimation (susceptible to objective factors such as ambient light, and subjective factors such as head deflection and occlusion), and at the same time improve the rate of model learning, the present invention provides a new solution, the specific composition of which will be explained in subsequent embodiments.

[0045] Please read first Figure 1 , Figure 1 A head posture positioning and detection method provided by an embodiment of the present invention can be applied to driving warning, classroom attention detection, children's behavior induction and other fields, including step S10, step S20 and step S30, which are described in detail as follows:

[0046] S10, obtaining an image set of the head posture to be measured, and performing standard brightness conversion on all images in the image set, wherein the image set includes an image set arranged in sequence in the image data, or an image set arranged in frame column sequence in the video data.

[0047] S20, use the person detection model and the head detection model to obtain the whole body position information Pa and the head position information Ph for all the images after the standard brightness conversion, input the whole body position information Pa and the head position information Ph into the pre-built deep learning convolutional neural network, and convert them to obtain the intermediate features Pa^ and Ph^ of the two position information.

[0048] S130, the intermediate features Pa^ and Ph^ of the two types of position information are fused according to a certain weight, and the head deflection Euler angles, namely the head deflection angles yaw, pitch, and roll, are calculated based on the fused feature P_site.

[0049] In a specific embodiment, the technical solution to achieve the above-mentioned invention object is: a head posture positioning and detection method, comprising the following steps:

[0050] (1) obtaining image or video data of the head posture to be measured, or using images in a standard data set, and performing data preprocessing operations commonly used in the field on the obtained person images, such as background cropping, correction alignment, rotation, mirroring, and decentralized cropping;

[0051] (ii) Using a person detection model based on deep learning (Yolo V3), detect the image preprocessed in step (i) to obtain the whole body position information Pa of the person in the image; using a head detection model based on deep learning (mmod_human_face_detector), obtain the head position information Ph of the person in the image; the position information Pa and Ph of each person in the same image are expressed in coordinate form, and the information format is as follows:

[0052]

[0053] Among them, n represents the number of the person to which the current coordinate belongs, Pn represents all the detected people in the image arranged in sequence, and (x1, y1, x2, y2) are the coordinates of the four vertices of the matrix box containing the whole body or head of the person.

[0054] ResNet 50 is used as the backbone network, and the position information Pa and Ph are used as feature inputs, which are output to the fully connected layer through the convolutional layer respectively. The intermediate position features output by the convolutional layer are denoted as Pa^ and Ph^.

[0055] (iii) The intermediate position features Pa^ and Ph^ obtained in step (ii) are fused according to a certain weight ratio as the final position features in the deep learning convolutional neural network (the whole body position information Pa^ of the person occupies a weight ratio of 0.4, and the head position information Ph^ of the person occupies a weight ratio of 0.6), and are integrated into the input of the network's fully connected layer. The specific calculation formula for position feature fusion is as follows:

[0056] P_site=(0.4Pa^⊙E+(0.6Ph^⊙E+α))

[0057] Among them, P_site is the character position information after weighted calculation, which is integrated into the ResNet network as an integrated feature; Pa^ and Ph^ are the character's whole body and head position information, ⊙ represents the dot product between elements; Pa^⊙E represents the unification of the intermediate position feature obtained after a convolution operation of Pa with the input dimension of the fully connected layer; Ph^⊙E represents the unification of the intermediate position feature obtained after a convolution operation of Ph with the input dimension of the fully connected layer; α is a 0 constant matrix, which is used to prevent Ph^ from being a null value due to the failure to detect head-related information in head detection.

[0058] (IV) According to the image brightness calculation standard of the international photoelectric protocol, calculate the brightness information of each frame of the video obtained in step (I), take three consecutive frames in the video, take the middle frame as the reference, and use the brightness information of each frame to replace the R, G, and B three-channel matrix of the traditional image (the brightness information of the first frame replaces the R channel, the brightness information of the second frame replaces the G channel, and so on) as the input of the model. The brightness calculation formula for each frame of the image is as follows:

[0059]

[0060] Wherein, formula (1) is the picture standard brightness calculation formula of the international photoelectric protocol, and formula (2) is the empirical formula for standard brightness calculation; A and Y both represent the picture pixel standard brightness matrix after calculation, and R, G, and B represent the matrix storing pixel color parameters.

[0061] Specifically, for the image or video data obtained in step (a), the data is first preprocessed, including but not limited to rotation, mirroring and decentralized cropping. After step (a), the original layer of the network is obtained. At this time, the traditional R, G, and B channels are not used as input, but three consecutive frames of brightness information are used as the input of the convolutional network, that is, step (d) is performed. In the convolutional layer of the network, the information features in steps (b) and (c) are used to accelerate the convergence of the model, and the two position features are fused before the fully connected layer, and the fused features are used as the input of the fully connected layer. Finally, the features processed by the pooling layer are used as output, and the predicted values ​​of the three deflection angles of the head posture are calculated using a deep learning method.

[0062] The ideas involved in each step of the present invention are generally introduced as follows:

[0063] If the image quality is low or the face of the person is blocked (making it difficult to detect the face), the position information of the person in step (ii) can be used to enhance the weight ratio of the person in the image during the training process, so that the model's attention converges near the person, thereby reducing the influence of irrelevant background factors. Furthermore, the use of the person's head position information can better reduce the influence of the rest of the person's body parts (that is, the body parts of the person except the head are also regarded as part of the 'environment'). In low-quality scenes (people wearing masks, glasses, or large-angle head deflection), the effect of head detection is poor. At this time, the weight ratio of the person's full-body position information Pa and the constant matrix α increases. The fused position information P_site obtained after step (iii) combines the fault tolerance of the full-body position information Pa (the coordinate information of Pa can fluctuate in a small range centered on the person) and the accuracy of the head position information Ph (converges on the head, more accurate). It should be noted that the constraint of position information makes the model's attention more focused during training and improves the learning rate of the model. Finally, the color information of each pixel in the image is replaced by a brightness value, which is a linear combination of R, G, and B. The input of three consecutive frames incorporates the temporal correlation between frames and removes the influence of light factors that are not related to head posture estimation.

[0064] Compared with the traditional head posture estimation method, the present invention has the following beneficial effects:

[0065] The present invention integrates the dual position information related to the target person of head posture estimation into the deep learning head posture estimation. Compared with the traditional head posture estimation method, it reduces the interference of background factors, so that the result of head posture estimation has a smaller deviation. The traditional neural network does not output the head posture estimation for undetected characters (due to large-scale occlusion or deflection, the characters are not recognized), while the present invention outputs the head posture estimation for each character in the image based on the combination of position information. In addition, the present invention uses brightness information instead of R, G, B channels to reduce the influence of light and color transformation, and uses position information to reduce the influence of background environment, so that the deep neural network has a faster convergence speed.

[0066] Figure 2 It is a module diagram of a head posture positioning and detection system provided in one embodiment of the present invention. The head posture positioning and detection system mainly includes an image set input module 100, a position information fusion module 200 and a deep learning module 300.

[0067] Among them, the image set input module 100 is used to obtain an image set of the head posture to be measured, and perform standard brightness conversion on all images in the image set. The image set includes an image set arranged in sequence in the image data, or an image set arranged in frame column order in the video data.

[0068] The position information fusion module 200 is used to obtain the whole body position information Pa and the head position information Ph of all images after standard brightness conversion using the person detection model and the head detection model, and input the whole body position information Pa and the head position information Ph into a pre-built deep learning convolutional neural network to convert the intermediate features Pa^ and Ph^ of the two types of position information.

[0069] The deep learning module 300 is used to fuse the intermediate features Pa^ and Ph^ of the two types of position information according to a certain weight, and calculate the head deflection Euler angle based on the fused feature P_site.

[0070] Figure 3 is a technical block diagram of a head posture and detection system provided by an embodiment of the present invention, such as Figure 3 As shown, the head posture and detection system includes an image set input module 100, a position information fusion module 200 and a deep learning module 300. The specific operation steps involved in each module are as follows:

[0071] Regarding the image set input module 100: First, obtain the image or video data of the person whose head posture is to be detected, and perform preprocessing operations. The specific operation process includes but is not limited to: Resize the image and adjust the size of the image to 224x224 pixels; arrange the images in the video in frame order (fps = 60), and arrange the images in the image set in order; it should be noted that there is no specific order requirement for the two steps of image size adjustment and image sorting. After completing the pre-processing, all images need to be converted, that is, the image is converted from the original R, G, B three-channel information to a single-channel image with only Y brightness information.

[0072] Figure 4 The standard brightness calculation principle diagram of the picture provided by the present invention is as follows: Figure 4 As shown, after arranging the pictures in the video in frame order, select three consecutive frames and store them in the list in the original order. The storage format is: L = (Pic1, Pic2, Pic3). Among them, the color information of each pixel of Pic1, Pic2, and Pic3 is stored in the three channels of R, G, and B, that is, stored in a 224x224x3 list; the picture brightness is calculated according to the standard brightness calculation formula. The optional calculation formula is as follows:

[0073] A=(0.2126*R)+(0.7152*G)+(0.0722*B) (a)

[0074] Y=0.299*R+0.587*G+0.114*B (b)

[0075] Y=sqrt(0.241*R^2+0.691*G^2+0.068*B^2) (c)

[0076] Among them, formula (a) and formula (b) are both standard brightness calculation formulas provided by the International Photoelectric Protocol. Due to the computer's own calculation mechanism, formula (c) can achieve faster calculation speed in actual operation, and the effect is similar, so formula (c) is used for calculation; after converting the original three channels into a single channel, a 224x224x1 list is obtained. Since three consecutive frames are taken, the brightness information of the three frames is used to replace the original three channels, that is, a 224x224x3 list is obtained again, and the dimension remains unchanged. The middle frame of the three consecutive frames is used as the brightness reference, and the two frames before and after reflect the change of brightness over time, and a correlation sequence between short-term frames is obtained. The brightness information of the three frames is stored in the list, and the storage format is (Y1, Y2, Y3). Y1, Y2, and Y3 are all calculated by formula (c).

[0077] Figure 5 The image standard brightness calculation flow chart provided by the present invention is Figure 4 The image standard brightness calculation principle diagram provided is combined to illustrate the method of calculating the standard brightness of the picture. The specific steps have been described in detail in the above method embodiment and will not be repeated here. Figure 5 The operation steps involved include the following:

[0078] S101, arranging the pictures in frame order;

[0079] S102, taking three consecutive frames of images;

[0080] S103, calculating pixel brightness;

[0081] S104, replacing the original R, G, and B dimensions;

[0082] S105, input neural network.

[0083] Regarding the position information fusion module 200: Figure 3As shown in FIG. 1 , the position information fusion module is located below the image set input module, and is used to receive the standardized image output by the image set input module, and input the fused position features to the deep learning module. The operation steps included in the position information fusion module are: first, receive the standardized image input in the form of (Y1, Y2, Y3). Assuming that there are n images in total, the storage sequence P = {x1, x2, x3, x4...xn} is obtained, where each image of x1, x2...xn is composed of (Y1, Y2, Y3). The input image is detected using the person detection model (Yolo V3), and the next step is performed according to the detection result: if a person is detected in the image, the next step is performed to detect the position of the detected person, including head position detection and whole body position detection. In the actual operation process, the person detection model (Yolo V3) can be used to realize the detection of the whole body position of the person, and most of the functions of the two are overlapped. Yolo v3 is composed of DBL (Darknetconv2d_BN_Leaky) basic components. During the detection process, k-means clustering is used for objects in the image, and multi-label classification is performed on the objects detected in the image.

[0084] If the person detection model does not detect a person, it is determined that the picture does not contain a person, and the next picture is detected, that is, the current detection sequence xn jumps to xn+1. If xn is the last picture in the sequence, the loop detection operation of the current position information fusion module ends and the deep learning module is executed. If the model detects a person, the Yolo v3 model is used to obtain the person's full body position information Pa, and the head detection model mmod_human_face_detector under Dlib is used to obtain the person's head position information Ph, in the following form:

[0085]

[0086]

[0087] Among them, the Pa sequence represents the whole body position information of {x1,x2,x3...xn}, and the Ph sequence represents the head position information of {x1,x2,x3...xn}. Each picture has a sequence that stores the position information of the person in the picture. The horizontal dimension is the total number of pictures n, and the vertical dimension is the number of people in the picture Pn. In the detection process of the Yolo v3 and mmod_human_face_detector models, if the head of the person is not detected, resulting in the head position information of the person being null, a 0 matrix α of the same dimension size is used to fill the vacant value. The reason for using a 0 matrix is ​​that there is no person in the picture, and the default yaw, pitch, and roll angles are 0°. Since the whole body position information and the person detection are the same model, when the whole body position information is null, the next picture is detected. If the last picture is detected, the detection is stopped.

[0088] Figure 6 A schematic diagram of the principle of fusing the whole body position information and the head position information provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown in the figure, the head position information and the whole body position information adopt a two stream structure, which are used as additional dimensions respectively. After the convolution training of the network, the intermediate position features Pa^ and Ph^ are obtained. The convolution body consists of four parts: layer1 (64x64), layer2 (128x128), layer3 (256x256), layer4 (512x512). After four-layer dimensional expansion, the intermediate features Pa^ and Ph^ are fused according to certain weights before the fully connected layer of the network. The calculation formula is: P_site = (0.4Pa∧⊙E+(0.6Ph∧⊙E+α)), Pa∧⊙E and Ph∧⊙E are the intermediate features of the two position information after the convolution layer, Pa has a weight of 0.4, Ph has a weight of 0.6, and α is the position 0 matrix.

[0089] About the deep learning module: The deep learning module receives the fused position features output by the position information fusion module and outputs three head deflection angles yaw, pitch, and roll after calculation. The main operations involved in the deep learning module are: after a convolution layer conv1, the input dimension is 3, the output dimension is 64, the convolution kernel size is set to 7, the step size is set to 2, and the padding is 3. It should be noted here that the input is not the traditional (R, G, B) dimension, but the brightness of three consecutive frames (Y1, Y2, Y3); the output 64-dimensional features are normalized, and the normalization method used is as follows:

[0090]

[0091] Among them, eps is a constant, set to 1e-5, gamma and beta are weight coefficients, y is the normalized output, and x is the input. After normalization, the linear correction unit ReLU is used for linear correction; at this time, the dimension size is still kept at 64, and the input of the linear correction unit passes through the maximum pooling layer (Max-pooling), the convolution kernel size is set to 3, the step size is set to 2, and the padding is 1; then four groups of upsampling operations are performed, that is, through layer1, layer2, layer3, layer4, as shown in Figure 3 As shown, the step size of each layer is set to 2; after four layers of upsampling, the dimension of the network becomes 512, which is put into the mean-pooling layer and then connected to the fully connected layer. The output of the fully connected layer is soft maxed, and the three deflection angles of the head posture, yaw, pitch, and roll, are obtained after calculation. The three deflection angles are displayed at the original image level. At this time, the operation steps of the head posture positioning and detection system combining position information and deep learning provided by the present invention are completed.

[0092] In order to better explain the system for positioning and detecting head posture in combination with position information provided by the present invention, a specific description is given below in conjunction with an embodiment.

[0093] Embodiment 1

[0094] Figure 7 A schematic diagram of head posture estimation in a classroom scene provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, in a classroom scene, the camera captures the student's class status, and obtains the student's corrected position information through the conversion between the camera coordinate system and the world coordinate system. The corrected position information is transmitted to the head posture positioning and detection system provided by the present invention to obtain the student's individual head position information and whole body position information. According to the obtained position information, the system further narrows the scope of head posture detection, so that the model's attention range is more focused. Finally, according to the results of deep learning training, the three deflection angles of the student's head in the current state, yaw, pitch, and roll, are obtained, and the orientation of the three angles of the student's head is represented by lines. The specific operation steps are as follows:

[0095] S201, camera detects people;

[0096] S202, obtaining position parameters;

[0097] S203, calculating the deflection angle;

[0098] S204, the character moves.

[0099] When the student's position moves, the camera captures the student's position at the new moment and provides feedback. The present invention then captures the student's position at the current moment and repeats the above steps to achieve the purpose of real-time detection.

[0100] Embodiment 2

[0101] Figure 8 A curve diagram of the student head deflection angle provided by an embodiment of the present invention, such as Figure 8 As shown, it reflects the head deflection of individual students within one minute. A one-minute student classroom video was selected, the video frame rate fps = 60, a total of 1800 pictures, the horizontal axis is the frames arranged in chronological order, the unit interval is 200 frames, and the vertical axis is the deflection angle, the unit interval is 5°; Figure 8 The deflection of the student at three angles within one minute is recorded, and three curves are used to represent the three deflection angles. It can be seen from the figure that the deflection angle is between -15° and 25°. In practical applications, this can be used as a basis for judging the student's class status, which is a scenario in which the present invention can be applied.

[0102] In order to prove the effect of the present invention, some specific data and their explanations involved in the experiment, the principle of the method adopted and its effect performance are provided, and the method proposed by the present invention is tested in a real scene. The experimental results show that the present invention has strong robustness for low-resolution images, large-area occlusion of the face and large-angle deflection of the head.

[0103] (1) Principle of the Invention

[0104] The present invention proposes a head posture positioning and detection method, which integrates the head position information and the whole body position information of the person into the convolutional neural network, so that the head posture estimation model has a faster convergence speed, and the additional position information assistance also improves the accuracy of head posture estimation; in addition, the present invention uses three consecutive frames of standard brightness to replace the traditional R, G, and B channels, optimizes the head posture estimation effect by adjusting the head edge detection position, and fully utilizes the whole body information of the person by feature fusion. Finally, the model effect is improved by selecting the best loss function. The above embodiments have introduced the principle method adopted by the present invention in detail, which will not be repeated here.

[0105] (2) Experimental process

[0106] Exploration on character position detection: In trying to obtain the position information of the characters in the image, the present invention has tried the following methods: 1. Pixel difference, 2. Background difference, 3. Deep learning. Among them, the principles of pixel difference and background difference are similar. Both use a background picture without characters and a picture with characters to make a difference to extract the characters in the image. The difference is that: pixel difference is from the perspective of pixels. After the difference, every change in the image will be obtained, while background difference goes a step further on the basis of pixel difference. By subtracting two pictures and setting a change threshold, the contour information of the characters in the image can be obtained. When the present invention uses background difference to process the image, the picture is first grayed, and then the background is subtracted to obtain a gray image with the background and foreground separated; then the separated image is binarized, the background is represented by 0 grayscale, and the foreground is represented by 255 grayscale; then noise reduction is performed to eliminate noise, the foreground picture boundary is obtained according to the grayscale value, and the character border is drawn; the experimental effect diagram is as shown in the figure. Fig. 9 shown.

[0107] Although background difference is simple and time-saving, when there are multiple characters in the scene, the background difference method cannot effectively separate the outlines of all characters, and this method requires additional background images, so it is not very applicable in real scenes. The present invention finally selects the deep learning character detection method (Yolo v3). Since this method is also based on deep learning, the character position features extracted by this method are easy to merge with the head posture detection method proposed in the present invention. By adjusting the object confidence threshold and selecting the character as the detection object, this method can even identify the character with only part of the body in the picture. The character position detection effect of this model in real scenes is as follows: Fig.10 shown.

[0108] Regarding head position detection research: The existing dlib_68 face detection model or human mod face detection model cannot meet the needs of real scenes for head detection under occlusion and deflection, such as Fig.11As shown. Based on the human mod model detection, the present invention adds a bounding box of the head position, so that the head position information of all characters in the picture can be detected. Like the character's whole body position information, the character's head position information is saved after model detection. In addition, the present invention also adds automatic adjustment of the character's head position border, by enlarging the bounding box around the character's head to 1.1 times or reducing it to 0.9 times of the original size, and then using the head detection model to detect the confidence of the face in the enlarged or reduced head border, and select the one with the highest confidence to save. Since different characters have different confidence results, this method is dynamic rather than static. The bounding box sizes of different characters' heads may be inconsistent, depending on the confidence of the model recognition. Experimental data show that the dynamic border transformation effect is due to the single border size. The head posture estimation is performed on the face with high confidence, making the estimation result more accurate.

[0109] from Fig.11 It can be seen that after adding bounding box information and automatically adjusting the frame, the present invention performs well in detecting people wearing masks, half-body figures, and people at a distance, and is more suitable for real scenes. Fig.12 (a) further explains the selection of the head frame position. In order to further demonstrate the effect of the fusion of the head position feature and the whole body position feature proposed by the present invention from the perspective of features, the present invention uses the attention heat map method to explore the attention area of ​​the model after the fusion features. Fig.12 As shown in (b), after being assisted by the dual position information features, the model's attention is obviously focused on the head area, which also explains why the model converges quickly. The model focuses on the head, the head information is effectively utilized, and the head posture estimation is more accurate.

[0110] like Fig.12 As shown, Figure (a) shows the dynamic change process of the head border adjustment of the present invention. Based on the size of the original image border, the model detects the content of the transformed border and selects the one with the highest confidence as the border of the current image. In Figure (b), from left to right are: the attention area of ​​the model without using position information; the attention area of ​​the model after using the whole body position information; the attention area of ​​the model after using the head information; the attention area of ​​the model after using the whole body + head position information.

[0111] (3) Experimental results

[0112] The present invention operates on a data set as follows: selecting images with a deflection angle of (-99°, 99°), and discarding images with a deflection angle exceeding this angle; uniformly adjusting the size of the images to 224x224; setting the batch size batch_size=32, and the worker thread num_workers=2; using 80% of the data set as a training set, 10% as a test set, and 10% as a verification set; selecting 80% of the images of the AFLW2000 data set and the 300W-LP data set as a training set, performing standard brightness conversion on the images in the training set, placing the converted images into an input layer of a network, and processing UPAN (video format data set), wherein each video in the UPAN data set has ten seconds, and 3 frames of images (a start frame, a middle frame, and an end frame) are selected per second for brightness conversion, each video has 30 images, a total of 10 characters, and each character has 12 unidirectional deflection videos, a total of 3600 images, and a total of 10 rounds of training.

[0113] The loss curves of the three angles during training are as follows Fig.13 As shown in the figure, the training loss curve of the AFLW2000 dataset is recorded as curve AF, the training loss curve of the 300W-LP dataset is recorded as LP, and the loss curve of the UPAN dataset is recorded as UP. The loss curve of each dataset is divided into three types: yaw, pitch, and roll. The loss function uses the mean square error loss (MSE). Fig. 9 It can be seen that in the UPAN dataset, the three angle deviation losses of head posture estimation finally converge to around 3°, among which yaw is 3.2°, pitch is 2.4°, and roll is 3.8°, with an average error MAE of 3.1°; in the 300W-LP dataset, the deviations of the three angles converge to around 4°, among which yaw is 2.7°, pitch is 3.9°, and roll is 4.3°, with an average error MAE of 3.6°; in the AFLW dataset, the deviations of the three angles converge to around 6°, among which yaw is 4.2°, pitch is 5.1°, and roll is 5.9°, with an average error MAE of 5.06°.

[0114] When conducting cross-dataset tests, the model trained with the UPAN dataset performed best, the model trained with the AFLW2000 dataset performed worst, and the model trained with the 300W-LP dataset was between the two. For model UP, the average estimation error of the three Euler angles was about 3.15 during cross-dataset testing; for model LP, the average deviation was about 3.5, and the average deviation of model AF was about 5.6 during cross-dataset testing. The cross-dataset head pose estimation errors of each model are detailed in Table 1, and the test results of the models in real scenarios are shown in Table 2. Fig.11 .

[0115] Table 1. Deflection loss of dataset

[0116]

[0117] (4) Results comparison

[0118] The present invention has achieved better performance in the test of public data sets. The better performance in cross-dataset verification proves the effectiveness of the present invention using dual position features.

[0119] (5) Ablation experiment

[0120] In order to explore the necessity of each optimization method of the head posture positioning and detection method proposed in the present invention, the necessity of the two optimization methods proposed in the present invention is explored by means of ablation experiments: 1. Dynamic head frame adjustment; 2. Standard brightness conversion. In the ablation experiment of the present invention, in order to make the data presentation concise and clear, the losses of yaw, pitch, and roll are added together and presented in the form of total loss; the ablation experiment is generally divided into two parts: the experimental group and the control group. The experimental group: (1) Experimental exploration including standard brightness conversion but not automatic frame adjustment, denoted as Y+Wb; (2) Experimental exploration including automatic frame adjustment but not standard brightness conversion, denoted as b+WY; (3) Experimental exploration including neither automatic frame adjustment nor standard brightness conversion, denoted as Wb+WY. Control group: the method of the present invention, that is, both are included, denoted as Y+b.

[0121] from Fig.14 It can be seen that the curve with automatic border adjustment confidence level and standard brightness conversion state (i.e., Y+b curve, the curve at the bottom) has the fastest convergence speed, and the final total deflection loss of this curve is the smallest; the difference in the starting points of the four curves is due to the partial randomness of the function parameters, the final loss of the Wb+WY curve is above 17°, the Y+Wb curve performs second, and the b+WY curve is third; this shows that the automatic adjustment of the border position information can play an auxiliary effect, and small-scale fine-tuning is better for some pictures, but not obvious for most pictures. Therefore, the auxiliary effect of automatic border position adjustment is not as obvious as the standard brightness conversion, and the standard brightness conversion is effective for every picture.

[0122] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A head posture positioning and detection method, characterized in that: The steps include: (1) Obtain an image set of the head posture to be measured, preprocess the acquired image set, and perform standard brightness conversion on all images in the preprocessed image set; the image set includes an image set arranged in sequence in the image data, or an image set arranged in frame column sequence in the video data; for the image data, directly perform standard brightness conversion on the R, G, and B channels of each image in the image set into a Y single-channel image; for the video data, obtain three consecutive frames of images in the image set, and perform standard brightness conversion on the R, G, and B channels of each image in the three frames of images to obtain three frames of brightness information (Y1, Y2, Y3) with short-term inter-frame correlation, wherein the brightness information Y2 of the middle frame image in the three frames of images is the reference brightness, and the brightness information Y1 and Y3 of the two frames before and after are the transformed brightness; the standard brightness conversion formula is: ； (2) Using a person detection model and a head detection model to obtain full body position information Pa and head position information Ph for all images after standard brightness conversion, the full body position information Pa and the head position information Ph are input into a pre-built deep learning convolutional neural network to obtain intermediate features Pa^ and Ph^ of the two position information; (3) The intermediate features Pa^ and Ph^ of the two position information are fused according to a certain weight, and the head deflection Euler angle is calculated based on the fused feature P_site; the fusion formula of the intermediate features Pa^ and Ph^ is: P_site = (0.4 Pa) ∧ ⊙E +(0.6Ph ∧ ⊙E +α)) Wherein, ⊙ represents the dot product between elements; α is a 0 constant matrix; Pa^⊙E represents the unification of the intermediate features obtained by the convolution layer operation of the deep learning convolutional neural network on the whole body position information Pa and the input dimension of the fully connected layer of the deep learning convolutional neural network; Ph ∧ ⊙E represents the unification of the intermediate features obtained after the convolutional layer operation of the head position information Ph by the deep learning convolutional neural network and the input dimension of the fully connected layer of the deep learning convolutional neural network.

2. The head posture positioning and detection method according to claim 1, characterized in that: The preprocessing includes one or more of rotation, mirroring, alignment correction and decentering.

3. The head posture positioning and detection method according to claim 1, characterized in that: The workflow of the deep learning convolutional neural network is: After one convolution layer conv1, one normalization layer bn1, linear unit correction ReLU, maximum pooling operation Max-pooling, and then four layers of upsampling operations, the results are average-pooled mean-pooling and then sent to the fully connected layer.

4. The head posture positioning and detection method according to claim 3, characterized in that: The deep learning convolutional neural network is trained using the UPAN dataset.

5. The head posture positioning and detection method according to claim 4, characterized in that: The loss function of the deep learning convolutional neural network adopts mean square error loss.

6. A head posture positioning and detection method as claimed in any one of claims 1 to 5, characterized in that: Applied in driving warning, classroom attention detection or children's behavior induction.

7. A head posture positioning and detection system, characterized in that: include: The image set input module is used to obtain an image set of the head posture to be measured, preprocess the obtained image set, and perform standard brightness conversion on all images in the preprocessed image set, wherein the image set includes an image set arranged in sequence in the image data, or an image set arranged in frame column sequence in the video data; for the image data, directly convert the R, G, and B channels of each image in the image set into a Y single-channel image by performing standard brightness conversion; for the video data, obtain three consecutive frames of images in the image set, and perform standard brightness conversion on the R, G, and B channels of each image in the three frames of images to obtain three frames of brightness information (Y1, Y2, Y3) associated with short-term frames, wherein the brightness information Y2 of the middle frame image in the three frames of images is the reference brightness, and the brightness information Y1 and Y3 of the two frames before and after are the transformed brightness; the standard brightness conversion formula is: ； A position information fusion module is used to obtain full body position information Pa and head position information Ph from all images after standard brightness conversion using a person detection model and a head detection model, input the full body position information Pa and the head position information Ph into a pre-built deep learning convolutional neural network, and convert them to obtain intermediate features Pa^ and Ph^ of the two types of position information; The deep learning module is used to fuse the intermediate features Pa^ and Ph^ of the two position information according to a certain weight, and calculate the head deflection Euler angle according to the fused feature P_site; the fusion formula of the intermediate features Pa^ and Ph^ is: P_site = (0.4 Pa) ∧ ⊙E +(0.6Ph ∧ ⊙E +α)) Wherein, ⊙ represents the dot product between elements; α is a 0 constant matrix; Pa^⊙E represents the unification of the intermediate features obtained by the convolution layer operation of the deep learning convolutional neural network on the whole body position information Pa and the input dimension of the fully connected layer of the deep learning convolutional neural network; Ph ∧ ⊙E represents the unification of the intermediate features obtained after the convolutional layer operation of the head position information Ph by the deep learning convolutional neural network and the input dimension of the fully connected layer of the deep learning convolutional neural network.

Images