Event-driven head posture estimation method, system, equipment and medium

Through the event-driven head attitude estimation method, the event space-time information fusion and motion information perception module are used to solve the accuracy and robustness problems of traditional technology in complex lighting and high-speed motion scenarios, and achieve higher head attitude Estimation accuracy and application capability.

CN118942156AActive Publication Date: 2024-11-12UNIV OF SCI & TECH OF CHINA

Patent Information

Application Number
CN202411010736.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-12
Estimated Expiration
2044-07-26

Smart Images

  • Figure CN118942156A_ABST
    Figure CN118942156A_ABST
Patent Text Reader

Abstract

The invention discloses a head posture estimation method, system and device based on event driving and a medium. In the scheme, based on event signal driving, the method, system and device are applied to head posture estimation tasks in various scenes by utilizing unique advantages of event signals; the method has a remarkable advantage of improving the pan-blurring condition of a traditional image in a head high-speed motion scene, and also has a good improvement effect on texture distortion of the traditional image in a complex illumination scene; through an event spatio-temporal information fusion module in the network, the influence caused by sparseness of event information in some scenes is reduced, the precision of head posture estimation in multiple scenes is improved, and through an event motion information sensing module in the network, attention is focused on a face key part which helps head posture estimation to be larger, so that the accuracy of head posture estimation is improved. The capability of paying attention to the effective part of sparse event motion information is effectively improved, and the application capability of the system in a complex environment is further enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an event-driven head posture estimation method, system, device and medium. Background Art

[0002] Head pose estimation is also an important research direction in the field of computer vision. Its main task is to accurately calculate the three rotation angles of the head (pitch, yaw and roll) by analyzing the face images in the scene sequence. Head pose estimation technology plays a key role in many application fields, including attention detection in driver monitoring systems, user interaction in virtual reality, emotion analysis in human-computer interaction systems, and identity recognition in security monitoring. With the development of deep learning technology, the accuracy and robustness of head pose estimation have been significantly improved. Existing methods can estimate head pose more accurately while improving the ability to adapt to fast motion and partial occlusion.

[0003] However, despite the remarkable achievements under slow speed and normal lighting conditions, the application of head pose estimation technology under complex lighting conditions and fast motion scenes still faces major challenges. Under such conditions, the quality of traditional images usually suffers from serious quality degradation. For example, there will be pan-blurring in high-speed motion scenes, distortion of head texture information under strong light conditions, loss of a large number of head details under weak light conditions, serious distortion of color information, insufficient information capture under conditions of dynamic lighting changes, etc., which will affect the accuracy and stability of the head pose estimation algorithm. More importantly, most of the current traditional image estimation algorithms rely on clear and high-quality image input as a prerequisite, which is often difficult to meet in actual scenes such as emergency braking, night driving, or indirect flashing of high and low beam lights. It is precisely because of the reliance on clear and high-quality head images that the application potential and actual effect of traditional image head pose estimation technology in various environments are greatly limited.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide an event-driven head posture estimation method, system, device and medium, which aims to utilize the high dynamic characteristics of event data and capture the high-speed motion characteristics of objects, so as to improve the accuracy of head posture estimation during high-speed motion and enhance the robustness under various lighting conditions.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A head posture estimation method based on event-driven, comprising:

[0008] Collect scene event streams at different resolutions, perform voxel gridding, and obtain label information;

[0009] Construct a head posture estimation network in multiple scenarios, which includes: event spatiotemporal information fusion module, feature extraction module, event motion information perception attention module and head posture estimation module;

[0010] The head posture estimation network is trained using the event data processed by voxel gridding, including: inputting the event data processed by voxel gridding, respectively extracting mean information and peak information through an event spatiotemporal information fusion module, and generating an event representation of fused spatial information after edge detection, extracting feature information through a feature extraction module, capturing head information through an attention mechanism through an event motion information perception attention module, obtaining feature information of event motion information perception, and then estimating the head posture information through a head posture estimation module; calculating a training loss function by combining the estimated head posture information with corresponding label information, and using the training loss function to train the head posture estimation network;

[0011] For the input event data after voxel gridding, the head pose information is estimated through the trained head pose estimation network.

[0012] A head posture estimation system based on event-driven, comprising:

[0013] A data collection and processing unit, used to collect scene event streams at different resolutions, perform voxel gridding processing, and obtain label information;

[0014] A network construction unit, used to construct a head posture estimation network in multiple scenarios, which includes: an event spatiotemporal information fusion module, a feature extraction module, an event motion information perception attention module, and a head posture estimation module;

[0015] A network training unit is used to train the head posture estimation network using the event data after voxel gridding, including: inputting the event data after voxel gridding, respectively extracting mean information and peak information through the event spatiotemporal information fusion module, and generating event representation of fused spatial information after edge detection, extracting feature information through the feature extraction module, using the attention mechanism of the event motion information perception attention module to capture head information, obtaining feature information of event motion information perception, and then estimating the head posture information through the head posture estimation module; calculating the training loss function by combining the estimated head posture information with the corresponding label information, and using the training loss function to train the head posture estimation network;

[0016] The head posture estimation unit is used to estimate the head posture information of the input event data after voxel gridding through the trained head posture estimation network.

[0017] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0020] It can be seen from the technical solution provided by the present invention that, based on event signal driving, by utilizing the unique advantages of event signals and applying them to head posture estimation tasks in various scenarios, this method has significant advantages in improving the general blur of traditional images in scenarios with high-speed head motion, and at the same time, it also has a good improvement effect on the texture distortion of traditional images in complex lighting scenarios; in the constructed network, the event spatiotemporal information fusion module is used to reduce the impact of the sparsity of event information in certain scenarios, and the accuracy of head posture estimation in multiple scenarios is improved; through the event motion information perception module, attention is focused on key parts of the face that are more helpful for head posture estimation, effectively improving the ability to focus on the effective part of sparse event motion information, and further enhancing the application ability of the system in complex environments; and, a supervised training method is adopted for training, and the high dynamic and deblurring characteristics of event information are utilized to improve the accuracy of head posture estimation in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 A flowchart of a head posture estimation method based on event-driven provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a head posture estimation network provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an event spatiotemporal information fusion module provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a single-layer structure of a RepVGG network provided in an embodiment of the present invention;

[0026] Figure 5A schematic diagram of the structure of a single-layer processing module in an event motion information perception attention module provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of an event-driven head posture estimation system provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0030] First, the terms that may be used in this article are explained as follows:

[0031] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0032] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0033] The following is a detailed description of an event-driven head posture estimation method, system, device and medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

[0034] Embodiment 1

[0035] The embodiment of the present invention provides a head posture estimation method based on event-driven, such as Figure 1 As shown, it mainly includes the following steps:

[0036] Step 1: Data collection and processing.

[0037] In the embodiment of the present invention, scene event streams at different resolutions are collected, voxel gridding is performed, and label information is obtained; specifically:

[0038] (1) Collect scene event streams E = {E0, E1, E2, E3, E4, ..., E T}, where E t Represents the event data corresponding to the t-th time slice, t=0, 1, 2, 3, 4, ..., T, and T represents the sequence number of the last time slice.

[0039] (2) Each time slice is processed into voxel grids to obtain event data after voxel grid processing, that is, voxel grid representation, and the event data E t The corresponding voxel grid representation is denoted by F.

[0040] (3) Preprocessing the voxel grid representation F to obtain the Euler angle label corresponding to each time slice, and converting it into a corresponding rotation matrix label, wherein the Euler angle label and the rotation matrix label are both the label information.

[0041] Step 2: Build a head pose estimation network for multiple scenarios.

[0042] In the embodiment of the present invention, the constructed head posture estimation network mainly includes: an event spatiotemporal information fusion module, a feature extraction module, an event motion information perception attention module and a head posture estimation module.

[0043] Step 3: Network training.

[0044] In an embodiment of the present invention, the event data after voxel gridding is used to train the head posture estimation network, including: inputting the event data after voxel gridding, respectively extracting mean information and peak information through an event spatiotemporal information fusion module, and generating an event representation of fused spatial information after edge detection, after the feature extraction module extracts feature information, the event motion information perception attention module uses an attention mechanism to capture head information, obtains feature information of event motion information perception, and then estimates the head posture information through a head posture estimation module; combining the estimated head posture information with the corresponding label information to calculate a training loss function, and using the training loss function to train the head posture estimation network.

[0045] Step 4: For the input voxel gridded event data, the head posture information is estimated through the trained head posture estimation network.

[0046] After completing the network training in step 3, the optimal head posture estimation neural network can be obtained, which is then used to process the voxel representations in different scenes to obtain the corresponding head posture information.

[0047] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0048] 1. Principle overview.

[0049] In the face of a series of challenges brought by high-speed motion and complex lighting scenes, the present invention proposes a head posture estimation method based on event-driven, where the event refers to event data. The method utilizes the high dynamic characteristics of event data and captures the high-speed motion of objects to construct a head posture estimation network including an event spatiotemporal information fusion module and an event motion information perception attention module. The event spatiotemporal information fusion module adopts a multi-channel information fusion strategy, which extracts peak information and mean information from event information and then fuses them with the original channel information to reduce the sparsity of its own content, reduce the impact of the sparsity of event information in certain scenes, and improve the accuracy of head posture estimation in multiple scenes; the event motion information perception module focuses on the key parts of the face (nose, mouth, etc.) that help the head posture estimation more, effectively improves the ability to focus on the effective part of sparse event motion information, and further enhances the application ability of the system in complex environments. Based on the method proposed by the present invention, the accuracy of head posture estimation technology for head posture estimation during high-speed motion can be improved, and the robustness under various lighting conditions can be improved to meet the actual needs of head-mounted VR (virtual reality) equipment or in-car driver scenes.

[0050] 2. Introduction of the plan.

[0051] 1. Collect data and process it.

[0052] In the embodiment of the present invention, a multi-scene event stream E={E0, E1, E2, E3, E4, ..., E T}, where E t Represents the event data corresponding to the t-th time slice, t=0, 1, 2, 3, 4, ..., T, and T represents the sequence number of the last time slice.

[0053] After that, each time slice is processed by voxel gridding to obtain the event data after voxel gridding, that is, voxel grid representation. tThe corresponding voxel grid representation is denoted as F. Exemplarily, the number of channels of the voxel grid representation may be 5.

[0054] Finally, the Euler angle label corresponding to each time slice is obtained through data preprocessing, and converted into the corresponding rotation matrix label, thereby obtaining the label information of each time slice. The preprocessing process involved in this part can refer to conventional technology, and the present invention will not be repeated.

[0055] 2. Build and train a head pose estimation network for multiple scenarios.

[0056] like Figure 2 As shown, a structural example of a head posture estimation network is provided, which mainly includes: an event spatiotemporal information fusion module, a feature extraction module, an event motion information perception attention module, and a head posture estimation module.

[0057] (1) Event spatiotemporal information fusion module.

[0058] The event spatiotemporal information fusion module includes: a mean information extraction module, a peak information module, an edge detection module and a fusion module.

[0059] The mean information extraction module is used to extract mean information from the event data after voxel gridding processing.

[0060] Extracting mean information from event data after voxel gridding mainly refers to performing a mean operation on the channel information represented by the voxel grid. The specific process is expressed as follows:

[0061]

[0062] Among them, F Mean (i,j) is the mean information of position point (i,j), F(i,j,k) is the k-th voxel information of position point (i,j), and B is the number of voxel information of a position point, that is, the number of channels represented by the voxel grid. That is, for the voxel grid representation F, the mean information F of each position point Mean It is obtained by summing and averaging the original voxel grid representation F.

[0063] The peak information extraction module is used to extract peak information from the event data after voxel gridding processing.

[0064] Extracting peak information from event data after voxel gridding mainly refers to performing peaking operations on channel information. The specific process is expressed as follows:

[0065]

[0066] Among them, F Max(i, j) is the peak information of the position point (i, j), that is, the peak information of each position point is obtained by taking the peak value (maximum value) of the original voxelized information F.

[0067] The edge detection module is used to perform edge detection using mean information and peak information respectively to obtain edge texture information; the edge detection includes horizontal and vertical edge detection.

[0068] In the embodiment of the present invention, an adaptive Sobel operator may be used for edge detection, and each Sobel operator is divided into horizontal and vertical directions for extracting certain edge texture information.

[0069] The fusion module is used to splice the event data, mean information, peak information and edge texture information after voxel gridding processing and fuse them with the event data after voxel gridding processing to obtain event representation of fused spatial information.

[0070] The relevant processing can be described as:

[0071]

[0072] Among them, F fusion is the event representation of fusion spatial information, Fusion is the fusion module, Concat means concatenation, and f sx 、f sy They represent the horizontal and vertical convolution processes in the adaptive Sobel operator, respectively. w represents mean information and peak information, w represents the averaging or maximization process (i.e. w represents Mean, Max), and F represents the voxel grid representation.

[0073] In the embodiment of the present invention, in order to simplify the form of the formula, the above formula uses F w To represent the mean information F Mean With peak information F Max The complete form in the brackets is

[0074] like Figure 3 As shown in the figure, a structural example of the event spatiotemporal information fusion module is provided. The left part is the entire structural example of the event spatiotemporal information fusion module, the middle part is the principle example of the adaptive sobel operator, and the right side is the structural example of the fusion module, which includes: batch normalization, 7×7 convolution, activation function, batch normalization, and 7×7 convolution.

[0075] (2) Feature extraction module.

[0076] In the embodiment of the present invention, the feature extraction module is mainly responsible for extracting feature information from the event representation fused with spatial information to obtain the event representation F enhanced by spatial information.l .

[0077] Exemplarily, the RepVGG network can be used as a feature extraction module. Figure 4 An example of a single-layer structure of the RepVGG network is provided, where the input of each layer is processed by 3×3 convolution and 1×1 convolution and then fused, and the same processing is continued in sequence as the output of one layer.

[0078] (3) Event motion information perception attention module.

[0079] In the embodiment of the present invention, the event motion information perception attention module includes: a plurality of layers of processing modules, each layer of processing modules includes: a first convolution layer, a grouped convolution module, a depth expansion convolution module, a batch normalization module, a second convolution layer and an attention product module arranged in sequence, and an activation function is arranged behind each convolution layer, such as Figure 5 A structural example of a single-layer processing module is provided.

[0080] The input feature map of the first layer processing module is the feature information extracted by the feature extraction module. The input feature map of each subsequent layer processing module is the feature map output by the previous layer processing module. In each layer processing module, the input feature map is convolved by the first convolution layer, and the global information of the event motion information is captured by the grouped convolution module. Then, the facial information captured from the global information is obtained through the depth expansion convolution module, the batch normalization module and the second convolution layer. In this process, the key information of the key parts of the event motion information (nose, mouth, etc.) is captured as much as possible; the facial information is multiplied by the input feature map through the attention product module, and the output result is superimposed with the input feature map as the output feature map, which is expressed as:

[0081]

[0082] Among them, F l is the input feature map, Attn is the captured facial information, is the output feature map, the superscript is the layer number, f conv Represents the convolution process.

[0083] Finally, the output feature map of the last layer of processing modules is the feature information X (feature attention part) perceived by the event motion information.

[0084] Figure 2 An example of a feature extraction module and an event motion information perception attention module is provided, that is, the two are stacked 4 times, and each feature map processed by the time motion information perception module must be processed by the feature extraction module and then enter the time motion information perception module. The output of the last processing module of the time motion information perception module is the feature attention part.

[0085] (4) Head posture estimation module.

[0086] In the embodiment of the present invention, the head posture estimation module may include: a global average pooling layer and an MLP (multi-layer perceptron). The feature information X perceived by the event motion information is subjected to global average pooling to obtain one-dimensional information of the feature map, and the head posture information is estimated by the MLP, which is expressed as:

[0087] R pred =f GS (W2(LeakyReLU(W1X+b1))+b2)

[0088] Among them, X is the feature information of event motion information perception, W1 and W2 are the weight parameters of the head posture estimation module, b1 and b2 are the bias parameters of the head posture estimation module (the above W and b are parameters in MLP), f GS is the mapping function from angle to matrix, R pred is the estimated head posture information, that is, the rotation matrix.

[0089] Angle to matrix mapping function f GS It can be expressed as:

[0090]

[0091] s2=p2-(q1·p2)q1

[0092] Among them, p1 and p2 are column elements before mapping, q1, q2, q3 are column elements after mapping, and q1, q2, q3, and s2 are all substitutes, mainly to simplify the form of the formula.

[0093] (5) Training loss function.

[0094] In the embodiment of the present invention, the loss function is calculated by combining the estimated head posture information with the corresponding label information. The label information includes: Euler angle label and rotation matrix label; the estimated head posture information is a rotation matrix, and the rotation matrix is ​​converted into Euler angle by a mapping function; the rotation matrix and the corresponding rotation matrix label are used to calculate the rotation matrix loss function L R , use the Euler angle and the corresponding Euler angle label to calculate the Euler angle loss function L E ; Combining the two, we get the training loss function, expressed as:

[0095] L total =L R +αL E

[0096] Among them, α is the weight coefficient of Euler angle loss, L total is the training loss function.

[0097] Among them, the rotation matrix loss function L R It is expressed as:

[0098]

[0099] Among them, R pred is the rotation matrix, i.e. the estimated head posture information, R gt is the rotation matrix label, tr() is the trace of the matrix, T is the transpose sign, and cos is the cosine function.

[0100] Euler angle loss function L E It is expressed as:

[0101]

[0102] Among them, |.| is the 1-norm symbol, f R→E (R pred ) represents the conversion of the rotation matrix into Euler angles, f R→E is the mapping function from the rotation matrix to the Euler angle, E gt It is the Euler angle label. The Euler angle contains three angles: pitch angle, yaw angle and roll angle. l refers to the lth angle in the Euler angle.

[0103] The mapping function f from the rotation matrix to the Euler angle R→E It can be expressed as:

[0104] f R→E (R pred )=(x,y,z)=(Pitch,Yaw,Roll)

[0105] x = arctan2(R 21 ,R 22 )

[0106] z = arctan 2(R 10 ,R 00 )

[0107] y = arctan 2(-R 20 ,sy)

[0108]

[0109] Among them, x corresponds to the pitch angle, y corresponds to the yaw angle Yaw, z corresponds to the roll angle Roll, and the symbol R with a subscript refers to the rotation matrix R pred The two-digit subscript value is the row number and column number of the matrix, for example, R 21 is the rotation matrix R pred The element at row 2 and column 1 of .

[0110] The head posture estimation neural network is trained using the gradient descent method, and the training loss function is calculated to update the network parameters. The training is stopped when the number of training iterations reaches a set number of rounds, thereby obtaining the optimal head posture estimation neural network.

[0111] 3. Head pose estimation.

[0112] The optimal head posture estimation neural network obtained through training is used to process the voxel representations in different scenes to obtain the corresponding head posture information. The processing flow involved in this part is the same as the training process, so it will not be repeated here.

[0113] The above solution provided by the embodiment of the present invention mainly achieves the following beneficial effects:

[0114] 1. This paper proposes an innovative head posture estimation design, which is driven by event signals and applied to head posture estimation tasks in various scenarios by utilizing the unique advantages of event signals. This method has significant advantages in improving the general blur of traditional images in scenes with high-speed head motion, and also has a good effect on improving the texture distortion of traditional images in scenes with complex lighting.

[0115] 2. The present invention has developed an event spatiotemporal information fusion module. The module adopts a multi-channel information fusion strategy, which reduces the content sparsity by extracting the peak information and mean information from the event information and fusing them with the original channel information, thereby reducing the impact of the sparsity of event information in certain scenarios and improving the accuracy of head posture estimation in multiple scenarios.

[0116] 3. The present invention also proposes an event motion information perception module, which focuses on key face parts (nose, mouth, etc.) that are more helpful for head posture estimation. This method effectively improves the ability to focus on the effective parts of sparse event motion information, and further enhances the application ability of the system in complex environments.

[0117] 4. The present invention better balances the convergence during training by combining rotation matrix loss and angle loss.

[0118] 5. The present invention adopts a supervised training method for training, and utilizes the high dynamic and deblurring characteristics of event information to improve the accuracy of head posture estimation in different scenarios.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0120] Embodiment 2

[0121] The present invention also provides an event-driven head posture estimation system, which is mainly used to implement the method provided in the above embodiment, such as Figure 6 As shown, the system mainly includes:

[0122] A data collection and processing unit, used to collect scene event streams at different resolutions, perform voxel gridding processing, and obtain label information;

[0123] A network construction unit, used to construct a head posture estimation network in multiple scenarios, which includes: an event spatiotemporal information fusion module, a feature extraction module, an event motion information perception attention module, and a head posture estimation module;

[0124] A network training unit is used to train the head posture estimation network using the event data after voxel gridding, including: inputting the event data after voxel gridding, respectively extracting mean information and peak information through the event spatiotemporal information fusion module, and generating event representation of fused spatial information after edge detection, extracting feature information through the feature extraction module, using the attention mechanism of the event motion information perception attention module to capture head information, obtaining feature information of event motion information perception, and then estimating the head posture information through the head posture estimation module; calculating the training loss function by combining the estimated head posture information with the corresponding label information, and using the training loss function to train the head posture estimation network;

[0125] The head posture estimation unit is used to estimate the head posture information of the input event data after voxel gridding through the trained head posture estimation network.

[0126] Considering that the main technical details involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0127] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0128] Embodiment 3

[0129] The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0130] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0131] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0132] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0133] The output device may be a display terminal;

[0134] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0135] Embodiment 4

[0136] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0137] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0138] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A head posture estimation method based on event-driven, characterized in that: include: Collect scene event streams at different resolutions, perform voxel gridding, and obtain label information; Construct a head posture estimation network in multiple scenarios, which includes: event spatiotemporal information fusion module, feature extraction module, event motion information perception attention module and head posture estimation module; The head posture estimation network is trained using the event data processed by voxel gridding, including: inputting the event data processed by voxel gridding, respectively extracting mean information and peak information through an event spatiotemporal information fusion module, and generating an event representation of fused spatial information after edge detection, extracting feature information through a feature extraction module, and then using an attention mechanism to capture head information through the event motion information perception attention module to obtain feature information of event motion information perception, and then estimating the head posture information through a head posture estimation module; calculating a training loss function by combining the estimated head posture information with corresponding label information, and using the training loss function to train the head posture estimation network; For the input event data after voxel gridding, the head pose information is estimated through the trained head pose estimation network.

2. The event-driven head posture estimation method according to claim 1, characterized in that: The collecting of scene event streams at different resolutions, performing voxel gridding processing, and obtaining label information includes: Collect scene event streams E={E0,E1,E2,E3,E4,...,E T }, where Et represents the event data corresponding to the t-th time slice, t = 0, 1, 2, 3, 4, ..., T, and T represents the sequence number of the last time slice; Each time slice is processed by voxel gridding to obtain event data after voxel gridding, that is, voxel grid representation. t The corresponding voxel grid representation is denoted as F; The voxel grid representation F is preprocessed to obtain the Euler angle label corresponding to each time slice, and converted into the corresponding rotation matrix label, wherein the Euler angle label and the rotation matrix label are both the label information.

3. The event-driven head posture estimation method according to claim 1, characterized in that: The event spatiotemporal information fusion module includes: a mean information extraction module, a peak information module, an edge detection module and a fusion module; wherein: The mean information extraction module is used to extract mean information from the event data after voxel gridding processing; The peak information extraction module is used to extract peak information from the event data after voxel gridding processing; The edge detection module is used to perform edge detection using mean information and peak information respectively to obtain edge texture information; the edge detection includes horizontal and vertical edge detection; The fusion module is used to splice the event data, mean information, peak information and edge texture information after voxel gridding processing and fuse them with the event data after voxel gridding processing to obtain event representation of fused spatial information.

4. The event-driven head posture estimation method according to claim 1, characterized in that: The event motion information perception attention module includes: a plurality of layers of processing modules, each layer of processing modules includes: a first convolution layer, a grouped convolution module, a depth expansion convolution module, a batch normalization module, a second convolution layer and an attention product module arranged in sequence, and an activation function is arranged behind each convolution layer; The input feature map of the first layer processing module is the feature information extracted by the feature extraction module. The input feature map of each subsequent layer processing module is the feature map output by the previous layer processing module. In each layer processing module, the input feature map is convolved by the first convolution layer, and then the global information of the event motion information is captured by the grouped convolution module. Then, the facial information captured from the global information is obtained through the depth expansion convolution module, the batch normalization module and the second convolution layer. The facial information is multiplied by the input feature map through the attention product module, and the output result is superimposed on the input feature map as the output feature map; Finally, the output feature map of the last layer of processing module is the feature information of event motion information perception.

5. The event-driven head posture estimation method according to claim 1, characterized in that: The head posture information estimated by the head posture estimation module is expressed as: R pred =f GS (W2(LeakyReLU(W1X+b1))+b2) Among them, X is the feature information of event motion information perception, W1 and W2 are the weight parameters of the head posture estimation module, b1 and b2 are the bias parameters of the head posture estimation module, and f GS is the mapping function from angle to matrix, R pred is the estimated head posture information, that is, the rotation matrix.

6. The event-driven head posture estimation method according to claim 1 or 5, characterized in that: The loss function calculated by combining the estimated head posture information with the corresponding label information includes: The label information includes: Euler angle label and rotation matrix label; The estimated head posture information is a rotation matrix, which is converted into Euler angles through a mapping function; Use the rotation matrix and the corresponding rotation matrix label to calculate the rotation matrix loss function L R , use the Euler angle and the corresponding Euler angle label to calculate the Euler angle loss function L E ; Combining the two, we get the training loss function, expressed as: L total =L R +αL E Among them, α is the weight coefficient of Euler angle loss, L total is the training loss function.

7. The event-driven head posture estimation method according to claim 6, characterized in that: Rotation matrix loss function L R It is expressed as: Among them, R pred is the rotation matrix, i.e. the estimated head posture information, R gt is the rotation matrix label, tr() is the trace of the matrix, T is the transpose sign, and cos is the cosine function; Euler angle loss function L E It is expressed as: Among them, |.| is the 1-norm symbol, f R→E (R pred ) represents the conversion of the rotation matrix into Euler angles, f R→E is the mapping function from the rotation matrix to the Euler angle, E gt It is the Euler angle label. The Euler angle contains three angles: pitch angle, yaw angle and roll angle. l refers to the lth angle in the Euler angle.

8. An event-driven head posture estimation system, characterized in that: include: A data collection and processing unit, used to collect scene event streams at different resolutions, perform voxel gridding processing, and obtain label information; A network construction unit, used to construct a head posture estimation network in multiple scenarios, which includes: an event spatiotemporal information fusion module, a feature extraction module, an event motion information perception attention module, and a head posture estimation module; A network training unit is used to train the head posture estimation network using the event data after voxel gridding, including: inputting the event data after voxel gridding, respectively extracting mean information and peak information through the event spatiotemporal information fusion module, and generating event representation of fused spatial information after edge detection, extracting feature information through the feature extraction module, using the attention mechanism of the event motion information perception attention module to capture head information, obtaining feature information of event motion information perception, and then estimating the head posture information through the head posture estimation module; calculating the training loss function by combining the estimated head posture information with the corresponding label information, and using the training loss function to train the head posture estimation network; The head posture estimation unit is used to estimate the head posture information of the input event data after voxel gridding through the trained head posture estimation network.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Head posture estimation method combined with YOLO-MobilenetV3 face detection

    CN113705521A

  • Head posture detection method and device and computer equipment

    CN114332911A

  • Head posture estimation method, device and equipment and storage medium

    CN117011929A

  • Head posture estimation method, device, equipment and vehicle

    CN118351516A

  • Systems and methods for pose estimation via radial voting based keypoint localization

    US20230316563A1

Cited By

  • Head posture estimation method based on facial feature enhancement of Laplace transformation

    CN121527851A