A method, device, equipment and medium for analyzing head attention in outdoor queue training

Abstract

The present invention relates to a method, device, equipment, and medium for analyzing head attention in outdoor queue training. The method comprises the following steps: obtaining a student's facial image region at three consecutive moments; calculating the facial spatial feature vector and the orientation posture 6D matrix of each moment frame based on the image region; obtaining the orientation posture Euler angle based on the orientation posture 6D matrix; calculating the average Euclidean distance of the facial spatial feature vector and the average difference of the orientation posture Euler angle for a consecutive time period frame; and determining whether attention is focused. When the average Euclidean distance of the facial spatial feature vector exceeds a preconfigured first threshold, or the average difference of the orientation posture Euler angle exceeds a preconfigured second threshold, the student is judged to be inattentive; otherwise, the student is judged to be focused. Compared with the prior art, the present invention has the advantages of continuous regression results and simple calculation.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a head attention analysis method, device, equipment and medium for outdoor queue training. Background Art

[0002] Formation training refers to standardized movement training performed according to instructor instructions. In traditional outdoor formation training, the observation, analysis, and evaluation of student training behavior are mostly based on traditional manual observation, resulting in subjective evaluation results. Furthermore, due to the large number of students in each team and the densely packed formations, manual observation and analysis cannot accurately capture every student's changes at every moment. Therefore, it is also impossible to accurately record incorrect movements to help students adjust their training posture. If computer-assisted AI-powered detection, analysis, and evaluation of teaching outcomes could be used, instructors could gain a more accurate understanding of student training effectiveness, including assessments of attention and movement standards.

[0003] The difficulties in this evaluation are: 1. In application scenarios, since most camera footage is captured from a side perspective, pose estimation is performed from a wide angle. 2. In computer vision, monocular head pose estimation methods are based on the forward direction relative to the camera, with the image center as the origin and the right-hand spiral rule as the positive direction. However, in actual training of moving queues, it is found that the head pose estimation parameters themselves are constantly changing, and fixed values ​​cannot be set to determine whether head deflection has occurred.

[0004] Traditionally, monocular head pose estimation is generally defined as a regression prediction task. Among the regression results, widely used pose representations include Euler angles and quaternions. However, both representations are inherently discrepant and cannot be fully one-to-one mapped to the original three-dimensional rotation matrix. Euler angles suffer from singularities, where the same pose can be mapped to the same pose using two different rotation directions and angles, leading to inaccurate predictions. While quaternions do not suffer from singularities, they do suffer from anti-pattern problems. Furthermore, mathematical proof suggests that any three-dimensional rotation representation smaller than five dimensions is discontinuous and difficult for neural networks to learn. This means that representations like Euler angles and quaternions are unsuitable for regression tasks within neural networks. Directly regressing a 3x3 three-dimensional rotation matrix requires regressing nine parameters, but only six variables are truly useful for prediction: the rotation angle in three directions and the offset, which is computationally complex. Furthermore, existing evaluations fail to account for variations in facial expression.

[0005] In summary, the current head pose estimation methods have many regression parameters, complex calculations, and cannot guarantee the continuity of the regression results. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method, device, equipment and medium for analyzing head attention in outdoor queue training with continuous regression results and simple calculation.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, a method for analyzing head attention during outdoor queue training is provided, comprising the following steps:

[0009] Get the student face image area at three consecutive moments;

[0010] According to the image area, the face spatial feature vector and the orientation posture 6D matrix of each time frame are calculated respectively, wherein the calculation process of the orientation posture 6D matrix is:

[0011] Obtaining a first image input tensor according to the image region, and performing convolution on the first image input tensor based on a pre-trained 6DHPENet custom head pose network model to obtain a first feature map;

[0012] Performing dimensionality reduction on the first feature map to obtain a first eigenvector, inputting the first eigenvector into a fully connected neural network layer whose output is 6-dimensional, to obtain a 6D matrix of orientation posture;

[0013] Obtaining Euler angles of the face orientation posture based on the 6D orientation posture matrix, wherein the Euler angles include a yaw angle of the face orientation posture rotating around the y-axis, a pitch angle of the face orientation posture rotating around the x-axis, and a roll angle of the face orientation posture rotating around the z-axis;

[0014] Calculating the average Euclidean distance of the face spatial feature vector and the average difference of the Euler angle of the orientation posture for a continuous time period frame;

[0015] Determine whether the attention is focused. When the average Euclidean distance of the facial spatial feature vector exceeds the preconfigured first threshold, or the average difference in the Euler angle of the orientation posture exceeds the preconfigured second threshold, it is determined that the attention is not focused. Otherwise, it is determined that the attention is focused.

[0016] Furthermore, the specific calculation process of the face space feature vector is:

[0017] Obtain a second image input tensor according to the image region, and perform convolution on the second image input tensor based on a pre-trained Resnet50 residual neural network model to obtain a second feature map;

[0018] Average pooling is performed on the second feature map, and the average pooled second feature map is converted into a face space feature vector.

[0019] Furthermore, the size of the first image input tensor is 224*224, the size of the second image input tensor is 224*224, the size of the first feature map is 1280*1*1, the size of the first feature vector is 1280*1, the second feature map is 2048 1*1 feature maps, and the size of the face space feature vector is 2048*1.

[0020] Furthermore, obtaining the Euler angle of the orientation posture based on the orientation posture 6D matrix is ​​specifically as follows: the orientation posture 6D matrix is ​​subjected to Gram-Schmidt orthogonal transformation to obtain a 3*3 rotation matrix, and the Euler angle of the orientation posture is calculated based on the rotation matrix.

[0021] Furthermore, the 6D matrix of the orientation posture is transformed through the Gram-Schmidt orthogonal transformation to obtain a 3*3 rotation matrix. The specific calculation process is:

[0022]

[0023] in, is the 6D matrix of the orientation posture, is the rotation matrix, and is the 3×1 column vector that constitutes the 6D matrix of the orientation posture, and is the 3×1 column vector that constitutes the rotation matrix, is the middle vector.

[0024] Furthermore, the calculation expression of the average difference of the Euler angle of the heading posture in a continuous time period frame is:

[0025]

[0026] in, is the average difference of the Euler angles of the orientation posture at the i-th moment, is the Euler angle of the orientation posture at the i-th moment, is the Euler angle of the orientation posture at the i-1th moment, is the Euler angle of the face orientation at the i-2th moment, and a is expressed as one of the yaw angle yaw of the face orientation around the y-axis, the pitch angle pitch of the face orientation around the x-axis, and the roll angle roll of the face orientation around the z-axis.

[0027] Furthermore, the expression for judging whether attention is focused is:

[0028]

[0029] in, is the average Euclidean distance of the face space feature vector, is the orientation pose 6D matrix, d1 is the preconfigured first threshold, and d2 is the preconfigured second threshold.

[0030] According to a second aspect of the present invention, there is provided a head attention analysis device for outdoor queue training, comprising:

[0031] Camera 1, used to capture training images of the first area;

[0032] Camera 2, used to capture training images of the second area;

[0033] The face positioning and target matching tracking module includes a cross-border head target tracking unit. When the student exceeds the detection area corresponding to the first camera, it is necessary to switch to the detection area corresponding to the second camera.

[0034] The head feature extraction and evaluation module includes a facial spatial feature vector calculation unit, a facial orientation posture estimation unit, and a head attention analysis unit. It calculates the similarity of spatial feature vectors and the difference in facial orientation posture between every three frames to judge the students' head attention concentration state during outdoor queue training.

[0035] According to a third aspect of the present invention, there is provided an electronic device for head attention analysis in outdoor queue training, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described is implemented when the processor executes the program.

[0036] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) A head posture estimation model that directly regresses the output value of the 6D rotation representation using deep learning is used to obtain the Euler angle of the orientation posture according to the 6D orientation posture matrix, which ensures the continuity of the regression result and replaces the 9 parameters required for direct regression of the 3x3 three-dimensional rotation matrix, thus reducing the amount of calculation.

[0039] (2) In combination with the face spatial feature vector, in addition to considering the influence of head posture orientation on head attention, the influence of facial expression changes on head attention is also considered, which improves the accuracy of attention analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the method of the present invention;

[0041] Figure 2 A schematic diagram of the software solution module structure of an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of the face positioning and target tracking and matching software according to an embodiment of the present invention;

[0043] Figure 4 This is a flowchart of the head feature extraction and attention analysis software of an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of the hardware structure of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] like Figure 1 As shown in FIG, the technical idea of ​​the head attention analysis method for outdoor queue training of the present invention is as follows: first, outdoor training scene images are collected to locate the face position and perform target matching and tracking; the face spatial feature vector and face orientation posture estimation are calculated respectively; the feature similarity and posture difference between frames in a time period are calculated; and finally, the head attention analysis results are output through normalized calculation.

[0047] Figure 2 The following is a schematic diagram of the software module structure of the preferred embodiment of the present invention. Figure 5 As shown in the hardware structure diagram of the present invention, Figure 2 、 Figure 3 、 Figure 4 The steps of the head attention analysis method for outdoor queue training of the present invention are described in detail:

[0048] like Figure 2 As shown in FIG, the analysis method steps can be divided into two main modules, S1 and S2. The S1 module is a face position positioning and target matching tracking module, and the S2 module is a head feature extraction and attention analysis module.

[0049] like Figure 3 FIG. 1 is a flowchart of the face positioning and target tracking software of a preferred embodiment of the present invention, which specifically illustrates the control steps of the S1 face position positioning and target matching tracking module:

[0050] In step 101, the original image is cropped and pixel values ​​are normalized to generate a 640*640 image input tensor. The input image is convolved using the yolov5 network model with pre-trained head target detection parameters to regress the candidate face target area and confidence factor.

[0051] In step 102, a threshold C is set and the confidence factor of the candidate face target region is determined to be greater than C. If it is greater than or equal to C, the target region is retained; if it is less than C, the target region is discarded. After all candidate target regions are determined, the target region that meets the threshold is retained, and the process proceeds to step 103.

[0052] In step 103, the training area is known to be a rectangular area. The coordinates of the four corner points of the rectangle are pre-marked. The coordinates (x, y) of the center point of each target area are calculated to see if they are within the closed area of ​​the matrix. If yes, the process proceeds to step 104. If not, the target area is discarded until all target areas have been judged.

[0053] In step 104, the face rectangular area is cropped, with the upper left corner of the image as the origin, the horizontal direction as the x direction, the vertical direction as the y direction, the target center point position coordinates as (x, y), the width as w, the height as h, the expansion area width parameter as pad, and the corresponding coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the cropped area are:

[0054]

[0055] In step 105, the deepsort network model is used to extract features from the cropped target area, and the rectangular area corresponding to the head target area is used as the input tensor. Convolutional feature extraction is performed to obtain a 751*1 feature vector.

[0056] In step 106, the extracted feature vector is used according to the deepsort algorithm, and the Kalman filter algorithm is used to predict the position of the center coordinate of the target area at the next moment. Then, the Hungarian algorithm is used to associate and match the feature vectors of the previous moment and the current moment. If the identity does not exist, a new ID is generated. If the identity exists, the corresponding target identity ID is returned and step 107 is entered.

[0057] In step 107 , it is determined whether the current frame image and the previous frame image belong to the same camera area. If so, the process proceeds to step 108 ; if not, the process proceeds to step 110 .

[0058] In step 108 , it is determined whether the identity ID of the current target area exists in the previous frame. If so, the process proceeds to step 111 ; if not, the process proceeds to step 109 to calculate the relative position distance.

[0059] In step 110, the position information of the key points of the human skeleton is first extracted using the posture estimation network model. Assuming that the sets of human key points in the last frame of the previous area and the current frame of the next area are t and a respectively, D represents the distance between each corresponding key point, the posture similarity S is calculated, ε is -1, and the formula is:

[0060]

[0061] The matching pair with the greatest similarity is assigned as the target matching pair in the front and back regions, and the identity ID of the previous region is assigned to the matching target person in the new region. The Euclidean distance between the center of the target person's clavicle and the center of the target person's face is then calculated for secondary matching. The matching relationship between the current facial region and the target person is determined, and the corresponding identity ID is assigned to the facial region. The process then proceeds to step 108 for identity matching.

[0062] In step 110, the Euclidean distance between the target face center positions at the current moment and the previous moment is calculated, and the target area with the smallest distance is selected as the associated matching pair, thereby assigning the matching identity ID of the previous moment to the matching target at the current moment.

[0063] In step 111, the target is matched and stacked to generate matched face sequence data, and the S1 module process ends.

[0064] like Figure 4 As shown in FIG, a flowchart of the head feature extraction and attention analysis software of a preferred embodiment of the present invention is shown, which specifically illustrates the control steps of S2 head feature extraction and attention analysis:

[0065] In step 401, the matched face sequence data is resized and pixel values ​​are normalized to generate a unified 224*224 image input tensor Heads. The image tensor is convolved using the Resnet50 residual neural network model with pre-trained parameters to extract shallow and deep texture features. After the last convolution operation, the obtained feature map is average pooled to obtain 2048 1*1 feature maps, which are converted into a two-dimensional 2048*1 face space feature vector to obtain the face space feature vector of the i-th time frame. Generate spatial feature sequence data and proceed to step 402;

[0066] In step 402, the average Euclidean distance of the face spatial feature vectors at three consecutive moments i-1, i, i+1 is calculated.

[0067] In step 403, the matched face sequence data is resized and pixel values ​​are normalized to generate a unified 224*224 image input tensor Heads. The image tensor is convolved using the 6DHPENet custom head posture network model with pre-trained parameters to extract shallow and deep texture features to obtain a 1280*1*1 feature map. The dimensionality is reduced to obtain a 1280*1 feature vector, which is sent to a fully connected neural network layer with a 6-dimensional output to obtain the estimated 6D matrix of the orientation posture. Generate the 6D matrix of the orientation posture of the i-th time frame The real 3*3 rotation matrix can be obtained through the Gram-Schmidt orthogonal transformation Wherein, the conversion relationship between the two is defined as follows:

[0068]

[0069] The 3*3 rotation matrix Calculate the Euler angle θ of the face orientation yaw ,θ pitch ,θ roll .

[0070] In step 404, the orientation pose 6D matrix Convert to Euler angle representation value Where, is the yaw angle of the face orientation around the y-axis at the i-th moment, is the pitch angle of the face orientation around the x-axis, The rolling angle of the face orientation around the z-axis is obtained, thereby obtaining sequence data in each direction and proceeding to step 405;

[0071] In step 405, the average difference in Euler angles of the face orientation posture at three consecutive moments i-2, i-1, i is calculated. The formula is as follows:

[0072]

[0073] In step 406, a normalization calculation is performed to determine the head concentration state value of student S at time i. The formula is as follows:

[0074]

[0075] Wherein, d1 and d2 are thresholds preset in advance based on human experience for judging whether the user is focused. If the thresholds are exceeded, the user is judged as not focused; if the thresholds are lower than the thresholds, the user is judged as focused, and the process proceeds to step 407 .

[0076] In step 407, the head attention analysis result is output.

[0077] The embodiment further provides a head attention analysis device for outdoor student queue training, comprising:

[0078] a. Camera 1, for capturing training images of the first area;

[0079] b. Camera 2, used to capture training images of the second area;

[0080] c. The face positioning and target matching and tracking module includes a cross-border head target tracking unit. When a student exceeds the detection area corresponding to camera one, it is necessary to switch to the detection area corresponding to camera two. During the area switching process, the identity of the same student in both images must be matched.

[0081] d. The head feature extraction and evaluation module includes a facial spatial feature vector calculation unit, a facial orientation posture estimation unit, and a head attention analysis unit. This module calculates the similarity of the spatial feature vectors and the difference in facial orientation posture between every three frames. Finally, it combines the similarity and orientation posture difference to determine the student's head attention state during outdoor queue training.

[0082] An embodiment further provides an electronic device for head attention analysis in outdoor queue training, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0083] The embodiment further provides a computer-readable storage medium having a computer program stored thereon, which implements the above method when the program is executed by a processor.

[0084] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A head attention analysis method for outdoor queue training, characterized in that: The following steps are involved: Get the student face image area at three consecutive moments; According to the image area, the face spatial feature vector and the orientation posture 6D matrix of each time frame are calculated respectively, wherein the calculation process of the orientation posture 6D matrix is: Obtaining a first image input tensor according to the image region, and performing convolution on the first image input tensor based on a pre-trained 6DHPENet custom head pose network model to obtain a first feature map; Performing dimensionality reduction on the first feature map to obtain a first eigenvector, inputting the first eigenvector into a fully connected neural network layer whose output is 6-dimensional, to obtain a 6D matrix of orientation posture; Obtaining Euler angles of the face orientation posture based on the 6D orientation posture matrix, wherein the Euler angles include a yaw angle of the face orientation posture rotating around the y-axis, a pitch angle of the face orientation posture rotating around the x-axis, and a roll angle of the face orientation posture rotating around the z-axis; Calculating the average Euclidean distance of the face spatial feature vector and the average difference of the Euler angle of the orientation posture for a continuous time period frame; Determine whether the attention is focused. When the average Euclidean distance of the facial spatial feature vector exceeds a preconfigured first threshold, or the average difference in the Euler angle of the orientation posture exceeds a preconfigured second threshold, it is determined that the attention is not focused. Otherwise, it is determined that the attention is focused. The specific calculation process of the face space feature vector is: Obtain a second image input tensor according to the image region, and perform convolution on the second image input tensor based on a pre-trained Resnet50 residual neural network model to obtain a second feature map; Performing average pooling on the second feature map, and converting the average pooled second feature map into a face space feature vector; The Euler angle of the orientation posture is obtained based on the 6D orientation posture matrix. Specifically, the 6D orientation posture matrix is ​​transformed into a 3*3 rotation matrix by Gram-Schmidt orthogonal transformation, and the Euler angle of the orientation posture is calculated based on the rotation matrix.

2. The head attention analysis method for outdoor queue training according to claim 1, characterized in that: The size of the first image input tensor is 224*224, the size of the second image input tensor is 224*224, the size of the first feature map is 1280*1*1, the size of the first feature vector is 1280*1, the second feature map is 2048 1*1 feature maps, and the size of the face space feature vector is 2048*1.

3. The head attention analysis method for outdoor queue training according to claim 2, characterized in that: The 6D matrix of the orientation posture is transformed into a 3*3 rotation matrix through the Gram-Schmidt orthogonal transformation. The specific calculation process is as follows: in, is the 6D matrix of the orientation posture, is the rotation matrix, and is the 3×1 column vector that constitutes the 6D matrix of the orientation posture, and is the 3×1 column vector that constitutes the rotation matrix, is the middle vector.

4. The head attention analysis method for outdoor queue training according to claim 1, characterized in that: The calculation expression of the average difference of the Euler angle of the heading posture in a continuous time period frame is: in, is the average difference of the Euler angles of the orientation posture at the i-th moment, is the Euler angle of the orientation posture at the i-th moment, is the Euler angle of the orientation posture at the i-1th moment, is the Euler angle of the face orientation posture at the i-2th moment, and a is expressed as one of the yaw angle yaw of the face orientation posture rotating around the y-axis, the pitch angle pich of the face orientation posture rotating around the x-axis, and the roll angle roll of the face orientation posture rotating around the z-axis.

5. The head attention analysis method for outdoor queue training according to claim 1, characterized in that: The expression for judging whether the attention is focused is: in, is the average Euclidean distance of the face space feature vector, is the orientation pose 6D matrix, d1 is the preconfigured first threshold, and d2 is the preconfigured second threshold.

6. A head attention analysis device for outdoor queue training, characterized in that: include: Camera 1, used to capture training images of the first area; Camera 2, used to capture training images of the second area; The face positioning and target matching tracking module includes a cross-border head target tracking unit. When the student exceeds the detection area corresponding to the camera, it is necessary to switch to the detection area corresponding to camera 2; The head feature extraction and evaluation module includes a facial spatial feature vector calculation unit, a facial orientation and posture estimation unit, and a head attention analysis unit. This module determines the student's head attention during outdoor queue training by calculating the similarity of facial spatial feature vectors and the difference in facial orientation and posture between every three frames. The head feature extraction and evaluation module implements the outdoor queue training head attention analysis method described in any one of claims 1 to 5.

7. An electronic device for analyzing head attention during outdoor queue training, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images