Head posture detection method, device and computer equipment

Abstract

The present application discloses a head posture detection method, device and computer equipment, the method comprising obtaining an event timing signal and RGB video data during the activity of an object to be detected; filtering out an event data stream and a key RGB video stream at the moment when the head posture of the object to be detected changes from the event timing signal and the RGB video data respectively and performing frame processing to obtain an event image sequence and a key RGB image sequence; outputting the event image sequence and the key RGB image sequence to a trained fusion model, extracting event modal features and image modal features respectively and fusing them to obtain a head posture feature image of the object to be detected; predicting the head posture angle of the object to be detected according to the head posture feature image; the present invention uses visible light-event dual-modal images to perform head posture estimation, which can effectively filter and capture the moment when the head posture changes, and can also achieve accurate estimation under the conditions of unsatisfactory lighting conditions, partial occlusion, etc.

Description

Technical Field

[0001] The present application relates to the field of educational management technology, and more specifically, to a head posture detection method, device and computer equipment. Background Art

[0002] In the past, the evaluation of teachers' teaching quality or credibility mainly adopted the method of conducting questionnaire surveys on students. This method often consumes a lot of time and manpower, is inefficient, and is not timely, making it difficult for teachers to adjust their teaching behavior strategies in a timely manner. In addition, in classroom teaching scenarios, scientifically quantifying the identification of teachers' teaching behaviors through classroom videos faces many problems, such as: lighting changes, motion blur, complex backgrounds, etc.

[0003] Head posture estimation is a very important application in the field of computer vision and pattern recognition. It plays an important role in the study of human behavior and attention and has been applied to many different fields. Changes in head posture contain rich information such as agreement, disagreement, understanding, confusion, and surprise. Using head posture estimation to identify teachers' teaching behaviors can provide teachers and teaching managers with reflection and evaluation information, promote the improvement of teachers' comprehensive abilities, and play a vital role in the entire teaching activity.

[0004] The current mainstream head posture estimation method uses a deep learning model to extract key feature points from the collected face images. This method is easily affected by the quality of the face image, as well as external factors such as lighting and complex background. Therefore, the accuracy is generally not high and the generalization ability is poor. In addition, this method is greatly affected by the network model. The model's ability to extract effective features and overfitting will affect the final estimation results. Summary of the invention

[0005] In view of at least one defect or improvement need in the prior art, the present invention provides a head posture detection method, apparatus and computer equipment, which aims to solve the problems of low accuracy and poor generalization ability of head posture estimation existing in the prior art.

[0006] To achieve the above object, according to one aspect of the present invention, a head posture detection method is provided, which comprises the following steps:

[0007] Acquire event timing signals and RGB video data during the activity of the object to be detected;

[0008] The event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes are respectively selected from the event timing signal and the RGB video data, and frame processing is performed to obtain an event image sequence and a key RGB image sequence;

[0009] Output the event image sequence and the key RGB image sequence to the trained fusion model, extract event modal features and image modal features from the event image sequence and the key RGB image sequence respectively, and fuse them to obtain a head posture feature image of the object to be detected;

[0010] Predict the head posture angle of the object to be detected according to the head posture feature image.

[0011] Preferably, in the head posture detection method, the network fusion model has an image network branch, an event network branch and a fusion module;

[0012] The image network branch is used to extract image modality features from the key RGB image sequence, and includes a first convolutional network, a first residual network, a first pooling layer, and a first linear mapping layer connected in sequence;

[0013] The event network branch is used to extract event modality features from the event image sequence, and includes a second convolutional network, a second residual network, a second pooling layer, and a second linear mapping layer connected in sequence;

[0014] The fusion module acquires image modality features and event modality features and fuses them.

[0015] Preferably, in the above head posture detection method, the training process of the network fusion model is:

[0016] Performing distillation constraints on the image modality features and the event modality features, wherein the distillation constraints include soft constraints, hard constraints, and self-consistency constraints;

[0017] The soft constraint is configured to obtain the mid-term image modality feature output by the first residual network and the mid-term event modality feature output by the second residual network, and calculate the similarity between the marginal probability distribution and the joint probability distribution of the mid-term event modality feature and the mid-term image modality feature;

[0018] The hard constraint is configured to obtain a late image modality feature output by the first linear mapping layer and a late event modality feature output by the second linear mapping layer, and calculate a distance between the late event modality feature and the late image modality feature;

[0019] The self-consistency constraint is configured to respectively obtain the early event modal features and the late event modal features generated by the second network branch and calculate the similarity between the marginal probability distribution and the joint probability distribution of the two.

[0020] Preferably, in the above head posture detection method, the soft constraint is expressed as:

[0021]

[0022] Among them, P X , P Y They represent the marginal probability distribution of mid-term image modality features and mid-term event modality features respectively; P (X,Y) represents the joint probability distribution of mid-term image modality features and mid-term event modality features; D KL represents the KL divergence.

[0023] Preferably, in the above head posture detection method, the hard constraint is expressed as:

[0024]

[0025] Wherein, E represents the Euclidean norm; r=1 or 2, r=1 represents the L1 norm, and r=2 represents the L2 norm.

[0026] Preferably, in the above head posture detection method, the self-consistency constraint is expressed as:

[0027]

[0028] Among them, (x e ,x l ) represent the early event modal features and the late event modal features extracted from the event network branches respectively; P (Xe,Xl) represents the joint probability distribution of the modal features of early events and the modal features of later events; P Xe , P Xl Respectively represent the marginal probability distribution of the modal features of early events and the modal features of late events;

[0029] D JS represents JS divergence; Q represents a discriminator composed of multiple two-dimensional CNN layers, which is used to predict whether the input feature pair comes from the same sample; (x e ′,x l ′) represents the marginal probability distribution (P Xe ,P Xl ) features extracted from the

[0030] Preferably, in the above-mentioned head posture detection method, the loss function of the network fusion model includes image branch loss and event branch loss;

[0031] The image branch loss is the difference between the head posture prediction value and the true value;

[0032] The event branch loss is obtained by weighted summing of soft constraints, hard constraints and self-consistency constraints.

[0033] According to a second aspect of the present invention, there is also provided a head posture detection device, comprising:

[0034] An acquisition unit, used for acquiring event timing signals and RGB video data during the activity of the object to be detected;

[0035] A data preprocessing unit, used to filter out the event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes from the event timing signal and the RGB video data respectively, and perform frame processing to obtain an event image sequence and a key RGB image sequence;

[0036] A fusion unit is used to extract event modal features and image modal features from the event image sequence and the key RGB image sequence respectively and fuse them to obtain a head posture feature image of the object to be detected;

[0037] The head posture estimation unit is used to predict the head posture angle of the object to be detected according to the head posture feature image.

[0038] Preferably, the head posture detection device further comprises a data acquisition unit for acquiring event timing signals and RGB video data during the activity of the object to be detected, which comprises an event camera and a visible light camera;

[0039] The event camera is used to collect event timing signals during the activity of the object to be detected;

[0040] The visible light camera is used to collect RGB video data of the object to be detected during its activity.

[0041] According to a second aspect of the present invention, there is also provided a computer device, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the above-mentioned methods.

[0042] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0043] (1) The head posture detection method, device and computer equipment provided by the present invention use event-RGB dual mode to estimate head posture. The network automatically extracts abstract features of the image, which can effectively filter and capture the moment when the teacher's head posture changes, greatly reducing the amount of network training and increasing the timeliness of network training. Accurate estimation can also be achieved under conditions such as poor lighting conditions and partial occlusion.

[0044] (2) The head posture detection method, device and computer equipment provided by the present invention constrain the network fusion model through soft constraints, hard constraints and self-consistency constraints, maximize the mutual information between event samples and image samples, and enhance the correlation between the feature maps from the event network branch and the image network branch; in addition, it can also prevent the event network branch from overfitting and losing effective information during the distillation process; thereby enhancing the robustness and generalization ability of the network fusion model and improving the accuracy of head posture estimation.

[0045] (3) Based on the head posture data output by the detection scheme provided by the present invention, the teacher's teaching behavior is identified, providing objective data support for solving the evaluation of the teacher's teaching process in the era of educational big data, and promoting the continuous enhancement of the teacher's comprehensive ability during the entire teaching activity. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a head posture detection method provided in this embodiment;

[0048] Figure 2 A schematic diagram of the network structure of the network fusion model provided in this embodiment;

[0049] Figure 3 A schematic diagram of the teacher's nonverbal behavior system provided for this embodiment;

[0050] Figure 4 A logic block diagram of the head posture detection device provided in this embodiment;

[0051] Figure 5 This is a diagram showing the structure of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0054] Figure 1 is a flow chart of a head posture detection method provided in this embodiment, see Figure 1 , which comprises the following steps:

[0055] S1 obtains the event timing signal and RGB video data during the activity of the object to be detected;

[0056] In this embodiment, the object to be detected is taken as a teacher as an example to collect event timing signals and RGB video data of the teacher in teaching activities; in a specific example, event cameras and RGB cameras arranged on both sides of the classroom are used to collect event timing signals and RGB video data of the teacher's teaching in the classroom.

[0057] S2: filtering out the event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes from the event timing signal and the RGB video data respectively, and performing frame processing to obtain an event image sequence and a key RGB image sequence;

[0058] In this embodiment, the collected event timing data and RGB video data are arranged and unfolded on the same time axis, the fluctuation of the event stream signal is observed and analyzed, and the moments with event stream signals are selected and recorded. During the period without signals, they are discarded. The final teaching event data stream and key RGB video stream are decomposed frame by frame, and then face detection and other preprocessing are performed using the RetinaNet-based face detection framework. In order to better adapt to the field of view of the event camera, the teaching event image sequence and the key RGB image sequence of the teacher's head are uniformly cropped to 512×512 size.

[0059] S3 outputs the event image sequence and the key RGB image sequence to the trained fusion model, extracts event modal features and image modal features from the event image sequence and the key RGB image sequence respectively and fuses them to obtain a head posture feature image of the object to be detected;

[0060] In a specific example, the above-mentioned teaching event image sequence and key RGB image sequence are respectively input into the EDNET network fusion model. The event signal contains simple and accurate facial contour and texture features and the characteristics of no background influence, which are used to bridge and highlight the teacher's head posture features in the RGB image, and the teacher's Event-RGB head posture image sequence is obtained by fusion.

[0061] Figure 2 is a schematic diagram of the structure of the EDNET network fusion model provided in this embodiment, such as Figure 2 As shown, the network fusion model has an image network branch, an event network branch and a fusion module;

[0062] Among them, the image network branch is used to extract image modal features from the key RGB image sequence, which includes a first convolutional network, a first residual network, a first pooling layer and a first linear mapping layer connected in sequence; the first convolutional network extracts features from the key RGB image sequence and outputs an image feature map; the role of the first residual network is to avoid the decrease in the feature extraction ability of the network as the number of network layers increases; the first pooling layer is used to downsample the image features output by the first residual network, and finally linearly map the features through the first linear mapping layer to obtain image modal features. In this example, the first convolutional network includes two convolutional layers connected in sequence, and the first residual network includes four residual layers connected in sequence; it should be pointed out that the number of convolutional layers and residual layers here is not specifically limited and can be adjusted as needed.

[0063] The event network branch is used to extract event modal features from the event image sequence, and includes a second convolutional network, a second residual network, a second pooling layer, and a second linear mapping layer connected in sequence;

[0064] The second convolutional network extracts features from the event image sequence and outputs an event feature map; the role of the second residual network is to prevent the network's feature extraction capability from decreasing as the number of network layers increases; the second pooling layer is used to downsample the event features output by the second residual network, and finally linearly map the features through the second linear mapping layer to obtain event modal features. In this example, the second convolutional network includes two sequentially connected convolutional layers, and the second residual network includes four sequentially connected residual layers; it should be noted that the number of convolutional layers and residual layers here is not specifically limited and can be adjusted as needed.

[0065] Fusion Module( Figure 2(not shown) image modal features and event modal features are obtained and fused. For example, the fusion module is a softmax layer, and the concatenate operation is used to fuse the image modal features output by the image network branch and the event modal features output by the event network branch.

[0066] The network fusion model can extract the unique features of each modality from traditional RGB images and event stream data. Specifically, given a pair of samples from image and event data {im,ev}, its corresponding data distribution {T IM ,S EV} and its corresponding data stream input {T IM ,S EV}, the N network modules in the network fusion model can extract {T IM ,S EV}A series of feature maps {t k (im),s k (ev)} N k = 1. Define the distillation constraint C(t k (im),s k (ev)) is used as the loss function to guide the event network branch S EV Therefore, designing constraints becomes a key issue in the RGB-EVENT distillation fusion framework.

[0067] This solution uses three modules (soft constraint, hard constraint and self-consistency module) to solve this problem. Figure 2 , the input side of the soft constraint module is connected to the output side of the first residual network and the second residual network respectively; the input side of the hard constraint module is connected to the output side of the first linear mapping layer and the second linear mapping layer respectively; the first input side of the self-consistency module is connected to the output side of the second residual network, and the second input side is connected to the output side of the second linear mapping layer. That is, the soft constraint module is placed after the third residual layer ResBlock, the hard constraint module with L2 norm is placed after the regression result of the linear mapping layer, and the self-consistency module is set after the first residual layer ResBlock and after the linear mapping layer.

[0068] Therefore, in this embodiment, the training process of the network fusion model is: performing distillation constraints on the image modality features and the event modality features, wherein the distillation constraints include soft constraints, hard constraints, and self-consistency constraints;

[0069] The soft constraint is configured to obtain the mid-term image modality feature output by the first residual network and the mid-term event modality feature output by the second residual network, and calculate the similarity between the marginal probability distribution and the joint probability distribution of the mid-term event modality feature and the mid-term image modality feature; the soft constraint is expressed as:

[0070]

[0071] Among them, P X , P Y They represent the marginal probability distribution of mid-term image modality features and mid-term event modality features respectively; P (X,Y) represents the joint probability distribution of mid-term image modality features and mid-term event modality features; D KL Represents the KL (Kullback–Leibler) divergence.

[0072] {T IM ,S EV The feature maps in the early stage have considerable differences in channels and space, but these features contain a lot of structural information, which is valuable for guiding the event flow; to this end, this scheme adopts a distillation bridge between the early feature maps of the two, {x, y} represents the feature map {t(im), s(ev)} of a random pair of samples {im, ev}, and {X, Y} represents the data distribution map {t(IM), s(EV)} of the sample pair. Feature information is extracted from the image by maximizing the mutual relationship between the event features in the early stage and the image network. The information distillation method is able to transfer knowledge from the image without considering the pixel-level or channel-level misalignment of the feature maps from the two network branches. By maximizing the mutual information between event samples and image samples, the goal is to link event samples with their paired image samples with minimal uncertainty and enhance the correlation of feature maps from the event network branch and the image network branch.

[0073] The hard constraint is configured to obtain the late image modality feature output by the first linear mapping layer and the late event modality feature output by the second linear mapping layer, and calculate the distance between the late event modality feature and the late image modality feature; the hard constraint is expressed as:

[0074]

[0075] Wherein, E represents the Euclidean norm; r=1 or 2, r=1 represents the L1 norm, and r=2 represents the L2 norm.

[0076] {T IM ,S EV} are mostly modality-specific and have large differences in visual attributes. The late features close to the prediction layer are task-specific and have common high-level representations. Since the two modes share common task goals, this scheme adopts L1 or L2 norms as hard constraints for two reasons: First, the network fusion model framework provided by this scheme aims to improve the performance of the event network branch and utilizes a student network (simple model) with the same capacity as the teacher model (complex model). Ideally, the student network can accurately simulate the output of the teacher model. Therefore, L1 or L2 norms are selected as hard constraints to capture accurate late features from traditional images and transfer them to the student model. Second, since L1 or L2 norms can be applied to both high-level tasks and low-level tasks, the network fusion model framework is flexible in dealing with different tasks.

[0077] The self-consistency constraint is configured to obtain the early event modal features and the late event modal features generated by the second network branch respectively and calculate the similarity between the marginal probability distribution and the joint probability distribution of the two; The self-consistency constraint is expressed as:

[0078]

[0079] Among them, (x e ,x l ) represent the early event modal features and the late event modal features extracted from the event network branches respectively; P (Xe,Xl) represents the joint probability distribution of the modal features of early events and the modal features of later events; P Xe , P Xl Respectively represent the marginal probability distribution of the modal features of early events and the modal features of late events;

[0080] D JS represents the JS (Jensen–Shannon) divergence; Q represents a discriminator composed of multiple two-dimensional CNN layers, which is used to predict whether the input feature pair comes from the same sample; (x e ′,x l ′) represents the marginal probability distribution (P Xe ,P Xl ) features extracted from the

[0081] Since event cameras are extremely sensitive to changes in light intensity, lighting conditions, motion conditions, and even the temperature of the scene, the output of event cameras may contain specific noise distributions generated by different types of teaching scenes. Therefore, a module is needed to prevent the event network branch from overfitting and losing a lot of information during the soft distillation process. Specifically, during the early distillation process of the model, the student network may sacrifice a lot of useful knowledge from the event data to approximate the image corresponding objects, resulting in the obtained student representations being weakly relevant to the target task (for example, the learned features may only be related to the background or part of the object). This makes the event network branch tend to overfit trivial features from the input rather than the decisive features we expect for the downstream task. Therefore, a module is needed to prevent the event network branch from overfitting and losing a lot of information during the soft distillation process. To this end, this scheme introduces a self-consistency module to maximize the mutual information between the features learned from the early and later stages, thereby encouraging the student network to consider all parts of the input during the distillation process.

[0082] It should be pointed out that the early, mid-term and late features described in this application are only used to represent the features extracted from the network structures at different levels in the network fusion model and the extraction order, and are not used to limit the features output by the network structure at a certain fixed level. As a preferred example, the early event modal feature is the feature output by the first residual layer in the second residual network, the mid-term event modal feature is the feature output by the last residual layer in the second residual network, and the late event modal feature is the feature output by the linear mapping layer.

[0083] In this embodiment, the loss function of the network fusion model includes the image branch loss L Target and the event branch loss L EV ; Among them, the image branch loss L Target is the difference between the predicted value and the true value of the head posture; Event branch loss L EV By soft constraint L Soft , hard constraint L Hard and the self-consistency constraint L Self The weighted sum of the three is obtained.

[0084] S4 predicts the head posture angle of the object to be detected according to the head posture feature image.

[0085] In this embodiment, the head posture feature image output by the network fusion model is input into the head posture estimation model to predict and output the three-dimensional output vector of the teacher's head posture angle.

[0086] The head posture estimation model can use a graph convolutional neural network, and its training process includes:

[0087] a. Establish a key point graph network relationship and use the RetinaNet face detection framework used in the preprocessing process to mark 18 key points. These key points need to meet the requirements of robustness and excellence.

[0088] b. The teacher's face image with the key point graph network relationship constructed is input into the network. The signal is forward propagated through the network and passes through two densely connected EVA-GC modules. Considering that different key points have unequal effects on the final posture estimation results, and different edges have unique importance when predicting different directions, the weights of each edge and each key point should not be the same. Therefore, this scheme introduces the attention mechanism.

[0089] c. Finally, the extracted abstract features are classified through the softmax output layer. The final output result of the network is the teacher head posture estimation parameter. The error between the network output and the target label is calculated to determine whether there is convergence;

[0090] d. If converged, the training ends; if not, the error is propagated backwards, the parameters are updated layer by layer, and then go to step c.

[0091] In order to effectively manage classroom teaching, teachers' head direction will attract students' attention, encourage students, concentrate their attention or converge their movements, and also enable teachers to improve their teaching level and change their teaching methods during subsequent teaching. After obtaining the angle and changes of the teacher's head posture during the teaching process, the teacher's non-verbal behavior can be quantified and recorded based on the head posture angle and changes; for example: according to the changes in the head posture recognition results at different times, combined with the teacher's head rotation frequency, the parameters of the teacher's non-verbal behavior can be quantified and compared. Figure 3 The teacher's non-verbal behavior system shown in the analysis and evaluation of the teacher's affinity (looking up, looking straight, looking down), management (facing students, facing the blackboard) and feedback (nodding, shaking head) during the teaching process.

[0092] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above-mentioned embodiments, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0093] This embodiment provides a device for head posture detection, which can be implemented in software and / or hardware and can be integrated on a computer device; Figure 4 As shown, the device comprises:

[0094] A data acquisition unit, used to acquire event timing signals and RGB video data during the activity of the object to be detected, comprising an event camera and a visible light camera, wherein the event camera is used to acquire event timing signals during the activity of the object to be detected; and the visible light camera is used to acquire RGB video data during the activity of the object to be detected;

[0095] An acquisition unit, used for acquiring event timing signals and RGB video data during the activity of the object to be detected;

[0096] A data preprocessing unit, used to filter out the event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes from the event timing signal and the RGB video data respectively, and perform frame processing to obtain an event image sequence and a key RGB image sequence;

[0097] A fusion unit is used to extract event modal features and image modal features from the event image sequence and the key RGB image sequence respectively and fuse them to obtain a head posture feature image of the object to be detected;

[0098] The head posture estimation unit is used to predict the head posture angle of the object to be detected according to the head posture feature image.

[0099] The specific implementation of each function in the device can be found in the detailed description of the head posture detection method part, which will not be repeated here.

[0100] This embodiment also provides a computer device, see Figure 5 , which includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned head posture detection method. The specific steps are shown in Example 1 and will not be repeated here. In this embodiment, the types of processors and memories are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be a volatile memory, a non-volatile memory, or a combination thereof, etc.

[0101] The computer device may also communicate with one or more external devices (such as a keyboard, a pointing terminal, a display, etc.), may communicate with one or more terminals that enable a user to interact with the computer device, and / or communicate with any terminal (such as a network card, a modem, etc.) that enables the computer device to communicate with one or more other computing terminals. Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter.

[0102] This embodiment also provides a computer-readable medium, which stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of the head posture detection method. The types of computer-readable media include but are not limited to storage media such as SD cards, USB flash drives, fixed hard disks, and mobile hard disks.

[0103] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A head posture detection method, characterized in that: include: Acquire event timing signals and RGB video data during the activity of the object to be detected; The event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes are respectively selected from the event timing signal and the RGB video data, and frame processing is performed to obtain an event image sequence and a key RGB image sequence; Output the event image sequence and the key RGB image sequence to the trained network fusion model, extract event modal features and image modal features from the event image sequence and the key RGB image sequence respectively and fuse them to obtain a head posture feature image of the object to be detected; Predicting the head posture angle of the object to be detected according to the head posture feature image; Wherein, the network fusion model has an image network branch, an event network branch and a fusion module; The image network branch is used to extract image modality features from the key RGB image sequence, and includes a first convolutional network, a first residual network, a first pooling layer, and a first linear mapping layer connected in sequence; The event network branch is used to extract event modality features from the event image sequence, and includes a second convolutional network, a second residual network, a second pooling layer, and a second linear mapping layer connected in sequence; The fusion module acquires image modality features and event modality features and fuses them; The training process of the network fusion model is: performing distillation constraints on the image modality features and event modality features, wherein the distillation constraints include soft constraints, hard constraints, and self-consistency constraints; The soft constraint is configured to obtain the mid-term image modality feature output by the first residual network and the mid-term event modality feature output by the second residual network, and calculate the similarity between the marginal probability distribution and the joint probability distribution of the mid-term event modality feature and the mid-term image modality feature; The hard constraint is configured to obtain a late image modality feature output by the first linear mapping layer and a late event modality feature output by the second linear mapping layer, and calculate a distance between the late event modality feature and the late image modality feature; The self-consistency constraint is configured to respectively obtain the early event modal features and the late event modal features generated by the second network branch and calculate the similarity between the marginal probability distribution and the joint probability distribution of the two.

2. The head posture detection method according to claim 1, characterized in that: The soft constraint is expressed as: in, , They represent the marginal probability distributions of mid-term image modal features and mid-term event modal features respectively; Represents the joint probability distribution of mid-term image modality features and mid-term event modality features; express Divergence.

3. The head posture detection method according to claim 1, characterized in that: The hard constraint is expressed as: in, represents the Euclidean norm; or 2, Time Representative norm, represent norm; represents a pair of samples from image and event data; express The feature map of .

4. The head posture detection method according to claim 1, characterized in that: The self-consistency constraint is expressed as: in, They represent the early event modal features and the late event modal features extracted from the event network branches respectively; Represents the joint probability distribution of the modal features of early events and the modal features of later events; , Respectively represent the marginal probability distribution of the modal features of early events and the modal features of late events; express Divergence; Represents a discriminator composed of multiple two-dimensional CNN layers, used to predict whether the input feature pair comes from the same sample; Represents the marginal probability distribution The features extracted from .

5. The head posture detection method according to claim 1, characterized in that: The loss function of the network fusion model includes image branch loss and event branch loss; The image branch loss is the difference between the head posture prediction value and the true value; The event branch loss is obtained by weighted summing of soft constraints, hard constraints and self-consistency constraints.

6. A head posture detection device, characterized in that: include: An acquisition unit, used for acquiring event timing signals and RGB video data during the activity of the object to be detected; A data preprocessing unit, used to filter out the event data stream and the key RGB video stream at the moment when the head posture of the object to be detected changes from the event timing signal and the RGB video data respectively, and perform frame processing to obtain an event image sequence and a key RGB image sequence; A fusion unit is used to extract event modal features and image modal features from the event image sequence and the key RGB image sequence respectively and fuse them to obtain a head posture feature image of the object to be detected; A head posture estimation unit, used for predicting the head posture angle of the object to be detected according to the head posture feature image; Among them, the network fusion model has an image network branch, an event network branch and a fusion module; The image network branch is used to extract image modality features from the key RGB image sequence, and includes a first convolutional network, a first residual network, a first pooling layer, and a first linear mapping layer connected in sequence; The event network branch is used to extract event modality features from the event image sequence, and includes a second convolutional network, a second residual network, a second pooling layer, and a second linear mapping layer connected in sequence; The fusion module acquires image modality features and event modality features and fuses them; The training process of the network fusion model is: performing distillation constraints on the image modality features and event modality features, wherein the distillation constraints include soft constraints, hard constraints, and self-consistency constraints; The soft constraint is configured to obtain the mid-term image modality feature output by the first residual network and the mid-term event modality feature output by the second residual network, and calculate the similarity between the marginal probability distribution and the joint probability distribution of the mid-term event modality feature and the mid-term image modality feature; The hard constraint is configured to obtain a late image modality feature output by the first linear mapping layer and a late event modality feature output by the second linear mapping layer, and calculate a distance between the late event modality feature and the late image modality feature; The self-consistency constraint is configured to respectively obtain the early event modal features and the late event modal features generated by the second network branch and calculate the similarity between the marginal probability distribution and the joint probability distribution of the two.

7. The head posture detection device according to claim 6, characterized in that: It also includes a data acquisition unit, which is used to collect event timing signals and RGB video data during the activity of the object to be detected, and includes an event camera and a visible light camera; The event camera is used to collect event timing signals during the activity of the object to be detected; The visible light camera is used to collect RGB video data of the object to be detected during its activity.

8. A computer device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 5.

Images