A method for acquiring head-up status data based on multiple data streams

Abstract

The present invention discloses a head-up status data acquisition method based on multiple data streams. The method collects video information through a three-eye camera (three-dimensional laser scanner, RGB camera and long-wave infrared camera) installed in a classroom, and obtains a part of frames in the video as a data set for head posture estimation; pre-processes 3D point cloud data, and fuses RGB images and infrared images together; uses a single-stage target detection algorithm to perform target detection and cropping on faces in the image; inputs the cropped image data into a trained head posture recognition model to obtain head posture deflection data of each student; counts the head-up rate of students and the head-up duration of each student within a certain period of time, and converts them into visual data, which is fed back to teachers to help teachers adjust teaching strategies in real time and teach students in accordance with their aptitude in a targeted manner, thereby improving teaching quality and effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and recognition, and more specifically, to a method for acquiring head-up status data based on multiple data streams. Background Art

[0002] Face recognition technology is a biometric technology that describes the overall features and outlines of a face. It is a technology that identifies a person through pattern recognition. However, face recognition in static images often fails to meet demand, so the research focus has gradually shifted to the recognition and tracking of dynamic images such as videos.

[0003] In daily teaching activities, head-raising rate is a very important indicator for evaluating teaching quality. It can well reflect students' concentration and monitor teachers' course quality. Real-time detection of head-raising rate and statistics of students' head-raising time are of great significance for teachers to change teaching strategies in time and remind students in a targeted manner.

[0004] However, existing face recognition methods based on video streams have problems such as insufficient three-dimensional spatial information in two-dimensional images, visible light images are easily affected by lighting, image clarity is general, and face rotation easily affects face recognition accuracy. As a result, the overall accuracy of face recognition is not very high, and the accuracy of head-up status data including head-up rate needs to be improved. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a method for acquiring head-up status data based on multiple data streams, so as to improve the overall accuracy of face recognition and the precision of acquiring head-up status data.

[0006] To achieve the above object, the present invention provides a method for acquiring head-up status data based on multiple data streams, comprising the steps of:

[0007] Collecting a preset amount of image data within a preset time, each of the image data including 3D point cloud data, visible light images and infrared images;

[0008] Preprocessing each of the image data separately, that is, normalizing the 3D point cloud data, and fusing the visible light image and the infrared image;

[0009] Using a single-stage target detection algorithm to perform target detection and cropping on the human face in each of the preprocessed image data;

[0010] Inputting the cropped image data into the head posture recognition model respectively to obtain the head posture deflection data of each image data;

[0011] Based on the head posture deflection data, the head-raising state data including the head-raising duration and the head-raising rate are obtained; specifically, whether it is a head-raising behavior is determined according to the three posture angles of the head posture deflection data and the corresponding pre-set thresholds; and whether the number of head-raising detected within the preset time is greater than the set threshold, if greater, it is determined that the head is always in the head-raising state, and the head-raising duration is increased by the preset time; the total number of people who raise their heads within the preset time is counted and the head-raising rate is calculated accordingly.

[0012] Furthermore, the fusing of the visible light image and the infrared image specifically comprises the steps of:

[0013] Using a plurality of convolutional layers to extract extensive low-level features from the input visible light image and the infrared image;

[0014] The element maximum fusion rule is used to fuse the extensive low-level features of multiple inputs; the fusion formula of the element maximum fusion rule is: in, represents the jth feature map of the ith image extracted by the two-dimensional matrix convolution operation, is the fused feature map of the jth channel obtained by fusion, fuse i Represents the maximum fusion rule of elements;

[0015] Several convolutional layers are used to reconstruct the fused features to obtain a fused image.

[0016] Furthermore, the fusing of the visible light image and the infrared image further comprises the steps of:

[0017] Obtain a total loss value of a loss function including a mean square error and a perceptual loss, and make the total loss value as small as possible.

[0018] Furthermore, the normalization process of the 3D point cloud data is specifically as follows:

[0019] The number of 3D point clouds is set to 2048, and the generated 3D point cloud data is normalized so that the point coordinates are limited to [-0.5, 0.5]. The normalization formula is as follows:

[0020]

[0021] Among them, P aver is the average value of a single head point cloud, max{Len x ,Len y ,Len z} is the maximum length of the three coordinates in the three-dimensional space, p is the coordinate of the head point cloud to be normalized, P resare the coordinates of the normalized head point cloud.

[0022] Furthermore, the head posture recognition model includes a first network model, a second network model, an overlay layer and a classification regression network;

[0023] The first network model is used to extract a first feature vector from the input fused image;

[0024] The second network model is used to extract a second feature vector from the input 3D point cloud data;

[0025] The superposition layer is used to perform a superposition operation on the extracted first feature vector and the second feature vector;

[0026] The classification regression network is used to output the head posture deflection data for the superimposed feature vector; the three posture angles of the head posture deflection data include a head posture deflection pitch angle, a head posture deflection roll angle and a head posture deflection yaw angle.

[0027] Furthermore, the first network model includes a mixing block and a plurality of information conversion flows, each of which includes multiple stages; the first stage includes a depthwise separable convolution layer and a pooling layer, and the Sth stage includes a depthwise separable convolution layer and a transformer encoding layer; S≥2;

[0028] The first stage uses a depth-separable convolutional layer to extract a feature map from the input fused image. The depth-separable convolutional layer is composed of a depth-wise convolutional layer and a point-wise convolutional layer. The depth-wise convolutional layer applies a single filter to each input channel, and the feature map output by each channel is:

[0029]

[0030] Among them, the size of the convolution kernel K is F k ·F k ·C in , I represents the input of the deep convolutional layer, whose size is C in ·F m ·F n ;

[0031] In the Sth stage, the feature map obtained in the S-1th stage is input into the corresponding depth-separable convolutional layer to extract a higher-level feature map, and then the higher-level feature map is converted into a sequence map and input into the corresponding transformer encoding layer, and the global context information is captured by attention to extract a higher-level feature map;

[0032] The mixing block superimposes the feature vectors extracted from a plurality of information conversion flows to obtain the first feature vector.

[0033] Furthermore, the second network model includes a plurality of set abstraction layers, each of which includes a sampling layer, a grouping layer and a pointnet layer;

[0034] The first set abstraction layer is used to receive the normalized 3D point cloud data and extract features; wherein the sampling layer determines the centroid according to the farthest point sampling method; the grouping layer captures the relationship between the centroid and their neighboring points according to the spherical query search, and then inputs it into the corresponding pointnet layer for feature extraction;

[0035] The Mth set abstract layer is used to receive the centroid selected in the M-1th set abstract layer for sampling; group the transformed point cloud in the M-1th set abstract layer; then input the grouped point cloud into the corresponding pointnet layer for feature extraction, and finally obtain the second feature vector; M≥2.

[0036] Furthermore, the farthest point sampling method comprises the steps of:

[0037] The input point cloud data has N points, and a point P0 is randomly selected from these points as the initial point to obtain a sampling point set S = {P0};

[0038] Calculate the distance from all points to P0, form an N-dimensional array L, select the point with the largest distance as P1, and update the sampling point set S = {P0, P1};

[0039] Then put P N′-2 As the initial point, calculate all points to P N′-2 The distance, for each point P i , if to P N′-2 If the distance is less than L[i], update L[i] and select the largest point in array L as P N′-1 , and update the sampling point set S = {P0, P1…P N′-1}; N′≥3.

[0040] Furthermore, feature extraction in the pointnet layer includes the following steps:

[0041] The coordinates of the point cloud in the centroid neighborhood are converted into a local frame relative to the centroid point using the following formula:

[0042] By using relative coordinates and point features to capture the point-to-point relationship in the local area, the formula is as follows:

[0043]

[0044] where i = 1, 2, ..., K and j = 1, 2, ..., d, are the coordinates of the center of mass, are the point cloud coordinates in the centroid field; is the transformed point cloud coordinate;

[0045] By using relative coordinates and point features to capture the point-to-point relationship in the local area, the formula is as follows:

[0046]

[0047] Among them, γ and h represent the multi-layer perceptron network.

[0048] Furthermore, extracting features in the transformer coding layer includes the steps of:

[0049] The feature map extracted by the depthwise separable convolutional layer Convert to sequential information As the input of the transformer encoder, where A = H × W; use this input to calculate the value vector V ∈ R A×C , key vector K∈R A×C and query vector Q∈R A×C , the calculation formula is:

[0050]

[0051] Among them, h represents the attention head, and represents the parameter matrix; the output of each attention head is Then connect the output of each attention head, the formula is:

[0052]

[0053] The output of the multi-head attention is used as the input of the Add&Norm layer. The output of the previous multi-head attention mechanism is superimposed with the current input through the residual connection, and the superimposed data is normalized; then it is input into the feedforward fully connected module; the output feature map is used as the input of the Add&Norm layer, and the output of this layer is used as the input of the next transformer encoder.

[0054] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0055] (1) The present invention provides a three-eye camera device for collecting 3D point cloud data, RGB images and long-wave infrared images. Compared with two-dimensional images, 3D point cloud data contains information of the third dimension, which solves the problem of insufficient three-dimensional spatial information in two-dimensional images. RGB images provide texture details with high spatial resolution and clarity, but are easily affected by light. In contrast, long-wave infrared cameras are not affected by light. The collected video frame data simultaneously includes three components of multiple data streams, and the three components complement each other, thereby enriching the information source and improving the information density of the collected video, laying a good foundation for subsequent accurate posture recognition.

[0056] (2) The present invention provides a network model that takes two modal data as input, extracts feature vectors from them respectively, and superimposes the feature vectors using an add layer (overlay layer) to increase the amount of information in each channel and improve the accuracy of head posture estimation. The transformer encoder is used in the first network model, which uses attention to capture global context information, extract more powerful features, and improve the robustness of the network model.

[0057] (3) The present invention takes into account the objective evaluation method of head-up rate in teaching evaluation, and can obtain the head-up rate of students in the classroom teaching process by analyzing the video images, with high detection accuracy. This method can judge the changes in students' head-up situation during the teaching process through face recognition technology and generate corresponding charts. In addition, this method can effectively analyze the teaching characteristics of teachers and students in the smart learning environment, and provide an objective evaluation method for the establishment of a teaching research database and the evaluation of teaching quality. The application of this method solves the problem of the lack of objective evaluation standards for the head-up rate in teaching evaluation, and can effectively analyze the teaching characteristics of teachers and students in the smart learning environment, providing technical support for the establishment of a teaching research database. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flowchart of a method for acquiring student classroom head-up status data based on multiple data streams provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a three-eye camera device and its application provided by an embodiment of the present invention;

[0061] Figure 3A schematic diagram of the network structure of the image fusion model provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of the network structure of a head posture estimation model provided by an embodiment of the present invention;

[0063] Figure 5 A block diagram of a system (electronic device) suitable for implementing the method described above provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0065] The terms "first", "second" or "third" etc. in the specification, claims or the above-mentioned drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0066] like Figure 1 As shown, in one embodiment, a method for acquiring student classroom head-up status data based on multiple data streams mainly includes the following steps:

[0067] Step 1: Obtain N image data within the T1 time period, including 3D point cloud data, visible light images, and infrared images.

[0068] Specifically, Figure 2 As shown, this embodiment uses a three-dimensional laser scanner, a visible light camera (RGB camera), and a long-wave infrared camera fixed on both sides of the classroom walls to capture 10 images of three modalities within 1 minute.

[0069] Step 2: Preprocess the 3D point cloud data, visible light image and infrared image of each image data respectively; use a single-stage target detection algorithm to detect and crop the faces in the preprocessed images.

[0070] Step 2.1: Specifically, for 3D point clouds, the number of points in the head varies depending on the input image. When people's faces are facing the camera, more facial areas are included and the number of points increases. When the face is away from the camera, the facial area in the depth image decreases and the number of points decreases. This embodiment sets the number of 3D point clouds to 2048 through a control experiment.

[0071] The 3D point cloud data generated above is normalized so that the point coordinates are limited to [-0.5, 0.5], thereby eliminating the adverse effects caused by singular sample data; the formula corresponding to normalization is as follows:

[0072]

[0073] Len max =max{Len x ,Len y ,Len z}

[0074] Among them, P aver is the average value of a single head point cloud, max{Len x ,Len y ,Len z} is the maximum length of the three coordinates in the three-dimensional space, p is the coordinate of the head point cloud to be normalized, P res are the coordinates of the normalized head point cloud.

[0075] Step 2.2: Fuse the visible light image and the infrared image. Figure 3 As shown in the figure, the image fusion model includes three modules: feature extraction, feature fusion and image reconstruction.

[0076] Step 2.2.1: Feature extraction: First, this embodiment uses two convolutional layers to extract extensive low-level features from the input visible light image and infrared image;

[0077] Step 2.2.2: Feature fusion: The features extracted in step 2.2.1 are fused through the feature fusion module; in this embodiment, the element maximum fusion rule is used to fuse the convolution features of multiple inputs, and the fusion formula is:

[0078]

[0079] in, represents the jth feature map of the i-th image extracted by CONV2, The fused feature map of the jth channel is obtained by the feature fusion module, fuse i Represents the maximum fusion rule of elements.

[0080] Step 2.2.3: Image reconstruction (feature reconstruction): Input the fused features into two convolutional layers, reconstruct the fused features, and obtain a fused image.

[0081] In fact, the image fusion model that fuses visible light images and infrared images also includes a loss calculation module, which is used to measure the gap between the predicted value and the true value. The goal is to make the total loss value (T loss ) gets smaller and smaller (as small as possible).

[0082] Step 2.2.4: Calculate loss: This embodiment uses mean square error and perceptual loss as factors of the loss function. The perceptual loss is expressed as:

[0083]

[0084] in, Represent the predicted fusion image and the real fusion image respectively, C f ,H f ,W f Represent the number of channels, height and width of the feature map respectively;

[0085] The formula for mean square error is:

[0086]

[0087] in, Represent the predicted fusion image and the real fusion image respectively, H g ,W g Represent the height and width of the real fused image respectively.

[0088] The formula for total loss is:

[0089] T loss =B loss +P loss .

[0090] Step 2.3: Preferably, this embodiment uses the YOLOv5 algorithm (a single-stage target detection algorithm) to perform target detection and cropping on the faces in the preprocessed image.

[0091] Step 2.3.1: Change the size of the input image to 448×448 and then feed the image into the convolutional neural network.

[0092] Step 2.3.2: The convolutional neural network divides the input image into a 10×10 grid, and each cell is responsible for detecting faces whose center point falls in that cell.

[0093] Step 3: Input the cropped image data into the trained MixNet network model (head posture recognition model) to obtain the head posture deflection result data at a certain moment.

[0094] like Figure 4 As shown, the head posture recognition model includes a first network model, a second network model, an add layer (overlay layer) and a classification regression network; the first network model is named TransDsc network, and the second network model is named PointNet++ network.

[0095] The first network model is used to extract a first feature vector from the input fused image.

[0096] The second network model is used to extract a second feature vector from the input preprocessed 3D point cloud data.

[0097] The add layer is used to perform a superposition operation on the extracted first eigenvector and second eigenvector.

[0098] The classification regression network is used to output head posture estimation for the superimposed feature vector.

[0099] Step 3.1: In this embodiment, the PointNet++ network includes five set abstraction layers and a fully connected layer, where each set abstraction layer includes a sampling layer, a grouping layer, and a pointnet layer. The specific steps are as follows:

[0100] Step 3.1.1: The first set of abstraction layers is used to receive 3D point cloud data and extract features.

[0101] Step 3.1.1.1: Given a set of point clouds {x1,x2,…,x n}, the sampling layer determines the centroid according to the farthest point sampling method To reduce the redundancy of original data and improve the calculation efficiency of the algorithm, the farthest point sampling method includes the following steps:

[0102] Step 3.1.1.1.1: The input point cloud data has N points. A point P0 is randomly selected from these points as the initial point to obtain the sampling point set S = {P0}.

[0103] Step 3.1.1.1.2: Calculate the distances from all points to P0, construct an N-dimensional array L, select the point with the largest distance as P1, and update the sampling point set S = {P0, P1}.

[0104] Step 3.1.1.1.3: Then take P1 as the initial point and calculate the distance from all points to P1. For each point P i, if the distance to P1 is less than L[i], then update L[i], select the largest point in the array L as P2, and update the sampling point set S = {P0, P1, P2}.

[0105] Step 3.1.1.1.4: Repeat steps 3.1.1.1.2 and 3.1.1.1.3 until N′ sampling points are obtained.

[0106] Step 3.1.1.2: The input of the grouping layer is a point cloud data of size N×(d+C) and a set of centroid point clouds of size N′×d, where N is the number of point clouds, d is the dimension of the point cloud, and C is other feature points, such as normals; the grouping layer captures the point cloud contained in the spherical space of each centroid based on the spherical query search, so that the subsequent network can learn the local features of each point; the output is a set of point clouds of size N′×K×(d+C), where K is the number of points in the field of each centroid point.

[0107] Step 3.1.1.3: The output of the grouping layer is used as the input of the pointnet layer. The output is the extraction of local features of each centroid and its neighborhood, and the output size is N′×(d+C′).

[0108] Step 3.1.1.3.1: First, convert the coordinates of the point cloud in the centroid neighborhood into a local frame relative to the centroid point using the following formula:

[0109]

[0110] where i = 1, 2, ..., K and j = 1, 2, ..., d, are the coordinates of the center of mass, are the point cloud coordinates in the centroid field; is the transformed point cloud coordinate.

[0111] Step 3.1.1.3.2: Capture the point-to-point relationship of the local area by using relative coordinates and point features. The formula is as follows:

[0112]

[0113] Among them, γ and h represent the multi-layer perceptron network.

[0114] Step 3.1.2: The second set abstraction layer is used to receive the centroid selected in the first set abstraction layer for sampling; group the transformed point cloud in the first set abstraction layer; and then input the grouped point cloud into the pointnet layer for feature extraction;

[0115] Step 3.1.3: The third, fourth and fifth set abstraction layers are similar to the second set abstraction layer.

[0116] Step 3.2: In this embodiment, the TransDsc network includes a mixing block and two streams (information conversion streams), each of which contains three stages to provide fine-grained regression for predicting head posture; the network structure of the first stream includes [DSC r (16)-AvgPool(2×2)-DSC r (32)-AvgPool(2×2]-[DSCr32-DSCr32-TransEnc-AvgPool2×2]-[DSC r (32)-DSC r (32)-TransEnc], the network structure of the second stream contains [DSC t (16)-MaxPool(2×2)-DSC t (32)-MaxPool(2×2)]-[DSC t (32)-DSC t (32)-TransEnc-MaxPool(2×2)]-[DSC t (32)-DSC t (32)-TransEnc], where DSC r (c) = [DSC(3×3,c)-BN-ReLU] and DSC t (c) = [DSC(3×3,c)-BN-Tanh, c is the number of channels; TransEnc is composed of three transformer encoders, and the self-attention mechanism uses 8-head attention.

[0117] Step 3.2.1: In the first stage, a depth-wise separable convolutional layer is used to extract features. The depth-wise separable convolutional layer consists of a depth-wise convolutional layer and a point-wise convolutional layer. The depth-wise convolutional layer applies a single filter to each input channel, and the feature map output by each channel is:

[0118]

[0119] Among them, the size of the convolution kernel K is F k ·F k ·C in , I represents the input of the deep convolutional layer, whose size is C in ·F m ·F n .

[0120] The computational cost of a depthwise convolutional layer is:

[0121] CompCost DC =F k ·F k ·Cin ·F m ·F n

[0122] The point-wise convolutional layer creates a linear combination of the features generated by the depth-wise convolutional layer. The computational cost of the point-wise convolutional layer is:

[0123] CompCost DC =C in ·C o ·F m ·F n

[0124] Step 3.2.2: The BN layer normalizes the feature map output by the depthwise separable convolution layer and then inputs it into the ReLU layer or the Tanh layer to enhance the learning and representation capabilities of the network. The ReLu function is f(x)=max(0,x), and the Tanh function is f(x)=2sigmoid(2x)-1. The feature map obtained above is then input into the pooling layer to compress it, remove redundant information, and simplify the network complexity.

[0125] Step 3.2.3: The second stage consists of two depth-wise separable convolutional layers to further extract features from the feature maps output from the first stage; see Figure 4 ,The second stage in this embodiment has 3 transformer encoders that use the attention mechanism to capture global contextual information, extract more powerful features, and improve the robustness of the network model.

[0126] In a specific example, the feature map extracted by the depthwise separable convolutional layer is Convert to sequential information As the input of the transformer encoder, where A = H × W. Use the input to calculate the value vector V ∈ R A×C , key vector K∈R A×C and query vector Q∈R A×C , the calculation formula is:

[0127]

[0128] Among them, h represents the attention head, and represents the parameter matrix. The output of each attention head is Then connect the output of each attention head, the formula is:

[0129]

[0130] The output of the multi-head attention is used as the input of the Add&Norm layer. The output of the previous multi-head attention mechanism is superimposed with the current input through the residual connection, and the superimposed data is normalized; then it is input into the feedforward fully connected module; this module consists of two linear transformations, with a ReLU activation function in the middle, and the corresponding mathematical formula is:

[0131] FFN(x)=max(0,xW1+b1)W2+b2

[0132] The output feature map is then used as the input of the Add&Norm layer, and the output of this layer is used as the input of the next transformer encoder.

[0133] Step 3.2.4: Refer to Figure 4 , The third stage contains two depth-wise separable convolutional layers and three transformer encoders to further extract features from the feature maps of the second stage.

[0134] Step 3.2.5: Input the feature maps obtained from the two streams into the mixing block, and combine the two feature maps into one feature map by multiplying the elements.

[0135] Step 3.3: Input the feature maps extracted by the PointNet++ network model and the TransDsc network model into the add layer for superposition operation.

[0136] Step 3.4: Input the superimposed feature map into the classification and regression network, see Figure 4 The classification regression network includes a 1×1 convolution layer, an average pooling layer, and a fully connected layer. The 1×1 convolution layer linearly combines the features on different channels to perform dimensionality reduction processing; then it is input into the average pooling layer to compress it and remove redundant information; then it is input into the fully connected layer to predict the head posture deflection. This embodiment uses the mean square error MSE as the loss function, and its formula is:

[0137]

[0138] Among them, y i and represent the true value and the predicted value respectively.

[0139] Step 4: Determine whether it is a head-up behavior based on the three posture angles estimated by the head posture and the pre-set threshold; and determine whether the number of head-up detected in the T1 time period is greater than the set threshold. If greater, the head-up duration + T1; and count the total number of head-up people in the T1 time period and calculate the head-up rate.

[0140] Specifically, the head posture deflection of each student in the 10 images collected by the trinocular camera within 1 minute is counted. If the pitch (i.e., rotating the object around the X-axis) angle ∈ [0, 45°] and the roll (i.e., rotating the object around the Z-axis) angle ∈ [-60°, 60°] and the yaw (i.e., rotating the object around the Y-axis) angle ∈ [-45°, 45°], then it is considered that a student is in a head-up state; the head-up state of each student in the 10 images is counted. If the head-up state is greater than 5, it is considered that the student has been looking up in this 1 minute, and the head-up time of the student is increased by 1 minute; the total number of students who have been in the head-up state within this 1 minute is counted, and the total number of students who look up / the total number of students in the class is obtained, which is the head-up rate of the class in this time period.

[0141] Figure 5 A block diagram of a system or electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. Figure 5 The system or electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 5 As shown, the electronic device 1000 described in this embodiment includes: a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0143] In RAM 1003, various programs and data required for the operation of system 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0144] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0145] According to the method flow of the embodiment of the present disclosure, it can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, equipment, device, module or unit described above can be implemented by a computer program module.

[0146] The embodiments of the present invention further provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0147] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.

[0148] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.

[0149] The flowchart or block diagram in the accompanying drawings illustrates the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope of the present disclosure.

[0151] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for acquiring head-up status data based on multiple data streams, characterized in that: Includes steps: Collecting a preset amount of image data within a preset time, each of the image data including 3D point cloud data, visible light images and infrared images; Preprocessing each of the image data separately, that is, normalizing the 3D point cloud data, and fusing the visible light image and the infrared image; Using a single-stage target detection algorithm to perform target detection and cropping on the human face in each of the preprocessed image data; Inputting the cropped image data into the head posture recognition model respectively to obtain the head posture deflection data of each image data; Based on the head posture deflection data, the head-raising state data including head-raising duration and head-raising rate are obtained; specifically, whether it is a head-raising behavior is determined according to the three posture angles of the head posture deflection data and the corresponding preset thresholds; and whether the number of head-raisings detected within the preset time is greater than the set threshold, if greater, it is determined that the head is always in the head-raising state, and the head-raising duration is increased by the preset time; Counting the total number of people who raise their heads within the preset time and calculating the head-raising rate accordingly; The fusion of the visible light image and the infrared image specifically comprises the following steps: Using a plurality of convolutional layers to extract extensive low-level features from the input visible light image and the infrared image; The element maximum fusion rule is used to fuse the extensive low-level features of multiple inputs; the fusion formula of the element maximum fusion rule is: in, represents the jth feature map of the ith image extracted by the two-dimensional matrix convolution operation, is the fused feature map of the jth channel obtained by fusion, fuse i Represents the maximum fusion rule of elements; Use several convolutional layers to reconstruct the fused features to obtain a fused image; The normalization process of the 3D point cloud data is specifically as follows: The number of 3D point clouds is set to 2048, and the generated 3D point cloud data is normalized so that the point coordinates are limited to [-0.5, 0.5]. The normalization formula is as follows: Among them, P aver is the average value of a single head point cloud, max{Len x , Len y , Len z } is the maximum length of the three coordinates in the three-dimensional space, p is the coordinate of the head point cloud to be normalized, P res are the coordinates of the normalized head point cloud; The head posture recognition model includes a first network model, a second network model, an overlay layer and a classification regression network; The first network model is used to extract a first feature vector from the input fused image; The second network model is used to extract a second feature vector from the input 3D point cloud data; The superposition layer is used to perform a superposition operation on the extracted first feature vector and the second feature vector; The classification regression network is used to output the head posture deflection data for the superimposed feature vector; the three posture angles of the head posture deflection data include a head posture deflection pitch angle, a head posture deflection roll angle and a head posture deflection yaw angle; The first network model includes a mixing block and a plurality of information conversion flows, each of which includes multiple stages; the first stage includes a depthwise separable convolution layer and a pooling layer, and the Sth stage includes a depthwise separable convolution layer and a transformer encoding layer; S≥2; The first stage uses a depth-separable convolutional layer to extract a feature map from the input fused image. The depth-separable convolutional layer is composed of a depth-wise convolutional layer and a point-wise convolutional layer. The depth-wise convolutional layer applies a single filter to each input channel, and the feature map output by each channel is: Among them, the size of the convolution kernel K is F k ·F k ·C in , I represents the input of the deep convolutional layer, whose size is C in ·F m ·F n ; In the S stage, the feature map obtained in the S-1 stage is input into the corresponding depth-separable convolutional layer to extract a higher-level feature map, and then the higher-level feature map is converted into a sequence map and input into the corresponding transformer encoding layer, and the global context information is captured by attention to extract a higher-level feature map; The mixing block superimposes the feature vectors extracted from a plurality of information conversion flows to obtain the first feature vector.

2. The head-up status data acquisition method according to claim 1, characterized in that: The fusing of the visible light image and the infrared image further comprises the steps of: Obtain a total loss value of a loss function including a mean square error and a perceptual loss, and make the total loss value as small as possible.

3. The head-up status data acquisition method according to claim 1, characterized in that: The second network model includes a plurality of set abstraction layers, each of which includes a sampling layer, a grouping layer and a pointnet layer; The first set abstraction layer is used to receive the normalized 3D point cloud data and extract features; wherein the sampling layer determines the centroid according to the farthest point sampling method; the grouping layer captures the relationship between the centroid and their neighboring points according to the spherical query search, and then inputs it into the corresponding pointnet layer for feature extraction; The Mth set abstract layer is used to receive the centroid selected in the M-1th set abstract layer for sampling; group the transformed point cloud in the M-1th set abstract layer; then input the grouped point cloud into the corresponding pointnet layer for feature extraction, and finally obtain the second feature vector; M≥2.

4. The head-up status data acquisition method according to claim 3, characterized in that: The farthest point sampling method comprises the steps of: The input point cloud data has N points, and a point P0 is randomly selected from these points as the initial point to obtain a sampling point set S = {P0}; Calculate the distances from all points to P0, form an N-dimensional array L, select the point with the largest distance as P1, and update the sampling point set S = {P0, P1}; Then put P N′-2 As the initial point, calculate all points to P N′-2 The distance, for each point P i , if to P N′-2 If the distance is less than L[i], update L[i] and select the largest point in array L as P. N′-1 , and update the sampling point set S = {P0, P1...P N′-1 }; N′≥3.

5. The head-up status data acquisition method according to claim 3, characterized in that: Feature extraction in the pointnet layer includes the following steps: The coordinates of the point cloud in the centroid neighborhood are converted into a local frame relative to the centroid point using the following formula: where i = 1, 2, ..., K and j = 1, 2, ..., d, are the coordinates of the center of mass, are the point cloud coordinates in the centroid field; is the transformed point cloud coordinate; By using relative coordinates and point features to capture the point-to-point relationship in the local area, the formula is as follows: Among them, γ and h represent the multi-layer perceptron network.

6. The head-up status data acquisition method according to claim 1, characterized in that: Extracting features in the transformer encoding layer includes the following steps: The feature map extracted by the depthwise separable convolutional layer Convert to sequential information As the input of the transformer encoder, where A = H × W; use this input to calculate the value vector V ∈ R A×C , key vector K∈R A×C and query vector Q∈R A×C , the calculation formula is: Among them, h represents the attention head, and represents the parameter matrix; the output of each attention head is Then connect the output of each attention head, the formula is: The output of the multi-head attention is used as the input of the Add&Norm layer. The output of the previous multi-head attention mechanism is superimposed with the current input through the residual connection, and the superimposed data is normalized; then it is input into the feedforward fully connected module; the output feature map is used as the input of the Add&Norm layer, and the output of this layer is used as the input of the next transformer encoder.

Images