Head pose estimation method and apparatus

Abstract

The present disclosure relates to a head pose estimation method and device, the method comprising: acquiring multi-modal data of a target object in a target time period; processing visual data in the target time period along a time direction to obtain a target matrix, and performing angular velocity calculation based on the visual data and inertial data in the target time period to obtain joint angular velocity in each direction; inputting the joint angular velocity in each direction into an angular velocity cell network to obtain a first pulse sequence, and inputting the target matrix into a spatial view cell network to obtain a second pulse sequence; inputting the first pulse sequence into a first intermediary cell network to obtain a first electrical signal, and inputting the second pulse sequence into a second intermediary cell network to obtain a second electrical signal; and inputting the first electrical signal and the second electrical signal into a head orientation cell network to obtain a calculation result of a head pose of the target object. The present disclosure can realize high-precision, high-real-time and high-stability head orientation judgment in a high-speed scene.

Description

Technical Field

[0001] This disclosure relates to the field of brain-like intelligence technology, and in particular to a head pose estimation method and device. Background Technology

[0002] Neuromorphic intelligence involves interdisciplinary fields such as neuromorphic computing, multimodal data fusion, and posture perception. In high-dynamic application scenarios such as mobile robots, autonomous aerial vehicles, unmanned racing cars, and virtual reality, accurately acquiring the target's head orientation information is a core prerequisite for achieving autonomous navigation and stable interaction. However, the overall design of related technologies does not consider the extreme dynamic characteristics of high-speed rotating fields. Hardware selection, fusion strategies, and information processing architectures are all based on low-speed motion scenarios, resulting in problems such as visual modality failure, a sharp drop in fusion accuracy, and poor accuracy and stability of head orientation estimation at high speeds. This makes it difficult to meet the needs of core application scenarios such as robots, drones, unmanned racing cars, and virtual reality in high-speed environments. Summary of the Invention

[0003] In view of this, this disclosure proposes a head pose estimation method and apparatus.

[0004] According to one aspect of this disclosure, a head pose estimation method is provided, which involves acquiring multimodal data of a target object within a target time period, wherein the multimodal data refers to visual data and inertial data collected by a brain-like visual sensor mounted on the target object; processing the visual data within the target time period along the time direction to obtain a target matrix; and calculating the joint angular velocity in each direction based on the visual data and inertial data within the target time period; inputting the joint angular velocity in each direction into an angular velocity cell network to obtain a first pulse sequence representing the change of the joint angular velocity in each direction; and inputting the target matrix into a spatial view cell network to obtain... A second pulse sequence representing the pulse firing frequency of the head-facing neuron associated with the spatial view neuron, wherein the spatial view neuron refers to a computational unit in the spatial view cell network, and the head-facing neuron refers to a computational unit in the head-facing cell network; the first pulse sequence is input into a first intermediary cell network to obtain a first electrical signal representing the input energy of the head-facing neuron, and the second pulse sequence is input into a second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-facing neuron; the first electrical signal and the second electrical signal are input into the head-facing cell network to obtain the calculation result of the head posture of the target object.

[0005] Thus, this embodiment of the present disclosure integrates visual and inertial multimodal data from a brain-like visual sensor, constructs features through a target matrix and joint angular velocity, generates pulse sequences through an angular velocity cell network and a spatial view cell network, and combines energy and calibration signals processed by an intermediate cell network. The head orientation cell network can output high-precision head posture. This method simulates biological neural mechanisms and can achieve high-precision, high-real-time, and high-stability real-time accurate estimation of head posture, i.e., head orientation judgment, in high-speed rotating scenes. It has strong anti-interference ability and is suitable for dynamic scenes such as virtual reality and assisted driving.

[0006] In one possible implementation, the step of processing the visual data within the target time period along the time direction to obtain the target matrix includes: converting the visual data within the target time period into a first matrix and a second matrix according to preset event types, wherein the event types include brightness increase and brightness decrease, the first matrix represents the event frequency of brightness increase corresponding to each pixel coordinate within the target time period, and the second matrix represents the event frequency of brightness decrease corresponding to each pixel coordinate within the target time period; and performing grayscale conversion based on the first matrix and the second matrix to obtain the target matrix.

[0007] In this way, the embodiments of this disclosure transform dynamic visual data into structured spatiotemporal features. By separating two types of events, namely brightness increase and decrease, the event frequencies of each pixel within the target time period are statistically analyzed to form a first matrix and a second matrix. These matrices are then fused into a target matrix through grayscale conversion. This design retains the event-driven characteristics of brain-like visual sensors, capturing the temporal patterns of pixel-level brightness changes and achieving efficient integration with subsequent neural networks through matrix form. This provides an input foundation for head pose estimation that combines spatial resolution and temporal dynamics.

[0008] In one possible implementation, the method further includes: generating a target image based on the first matrix and the second matrix.

[0009] In this way, the embodiments of this disclosure transform abstract event frequency data into intuitive visual information, which not only preserves the spatiotemporal distribution of the original event characteristics, but also provides a visual basis for algorithm debugging and result verification.

[0010] In one possible implementation, the step of calculating the joint angular velocity in each direction based on visual data and inertial data within the target time period includes: calibrating the inertial data within the target time period to obtain the calibration angular velocity in each direction, and calculating the standard deviation of the angular velocity in the corresponding direction based on the calibration angular velocity in each direction; determining the event angular velocity and pixel matching ratio in each direction based on the visual data within the target time period; and determining the joint angular velocity in the corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity in each direction.

[0011] Thus, in this embodiment of the present disclosure, the inertial data is first calibrated to obtain the calibrated angular velocity in each direction and the standard deviation is calculated to quantify the noise level. Event angular velocity and pixel matching ratio are extracted from visual data. Finally, by fusing event angular velocity, pixel matching ratio, angular velocity standard deviation and calibrated angular velocity, a joint angular velocity with both dynamic response speed and measurement stability is generated, which effectively suppresses the defects of a single sensor and provides robust kinematic input for head posture estimation.

[0012] In one possible implementation, determining the joint angular velocity for a corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity for each direction includes: for each direction, calculating a first ratio based on the angular velocity standard deviation and a preset maximum standard deviation; calculating a first difference based on the preset standard deviation and the first ratio; and using the product of a preset first ratio and the first difference, plus the sum of a preset second ratio, as a first weight for the corresponding direction; calculating a second ratio based on the pixel matching ratio and a preset maximum matching ratio; and using the product of the first ratio and the second ratio, plus the sum of the second ratio, as a second weight; normalizing based on the first weight and the second weight for each direction to obtain a first normalized weight and a second normalized weight for each direction; determining the first angular velocity for the corresponding direction based on the first normalized weight and the calibration angular velocity for each direction, and determining the second angular velocity for the corresponding direction based on the second normalized weight and the event angular velocity; and using the sum of the first angular velocity and the second angular velocity for each direction as the joint angular velocity for the corresponding direction.

[0013] Thus, this embodiment dynamically adjusts the weight ratio of inertial data (calibration angular velocity) and visual data (event angular velocity) by using the standard deviation of angular velocity and the pixel matching ratio. After normalization, the weighted sum is used to obtain the joint angular velocity, which can achieve adaptive fusion of multi-source data. The weight allocation can be optimized in real time according to noise intensity and feature quality, effectively balancing dynamic response and measurement stability, significantly improving the accuracy of angular velocity calculation, and providing robust kinematic input for head pose estimation.

[0014] In one possible implementation, the method further includes: determining a first motion trend for each target pixel based on visual data within the target time period, and determining a corresponding second motion trend based on the joint angular velocity; if there are a preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be valid; if there are fewer than the preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be invalid, and head pose estimation is performed based on valid multimodal data within historical time periods or by re-acquiring multimodal data within a new target time period.

[0015] Thus, the embodiments of this disclosure improve data reliability through motion trend consistency verification. The first motion trend of the target pixel extracted from the visual data is compared with the second motion trend derived from the joint angular velocity. When the number of matching pixels reaches the standard, the data is determined to be valid. This can promptly identify sensor failure or feature failure scenarios. By enabling historical valid data or re-collecting, it avoids attitude estimation deviations caused by abnormal inputs, significantly enhances the robustness of the system in complex dynamic environments, and provides data quality assurance for continuous and stable head attitude tracking.

[0016] In one possible implementation, the method further includes: converting the target matrix into a target neuron state matrix according to a preset time encoding rule; calculating the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each preset spatial view template; determining the estimated head pose of the target object based on the head pose calculation result corresponding to the spatial view template with the highest similarity, wherein the estimated result is used to drive the attractor state transition in the spatial view cell network.

[0017] Thus, this embodiment introduces a spatial view template matching mechanism, which converts the target matrix into a neuron state matrix through time encoding, compares it with a preset spatial view template, and selects the template pose with the highest similarity as the estimation result. The estimation result can be used to guide the attractor state transition of the spatial view cell network, accelerate network convergence, and improve the initial accuracy of pose estimation. Especially in scenarios with blurred features or rapid motion, template matching can provide reliable pose priors and enhance the system's adaptability to complex environments.

[0018] In one possible implementation, the method further includes: if the comparison between the calculated head pose determined based on multimodal data within the target time period and the calculated head pose determined based on multimodal data within the previous time period meets a preset condition, and the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset first threshold, then the preset spatial view template is updated; if the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset second threshold, then the preset spatial view template is updated.

[0019] Thus, this embodiment of the present disclosure achieves dynamic updating of spatial view templates through dual-condition triggering. When the change in pose calculation results of two consecutive frames meets the preset conditions and the similarity between the current neuron state and all templates is lower than the first threshold, or when the similarity between all templates is lower than the second threshold, template updating is triggered. This can adapt to environmental changes, retain reliable prior knowledge, and learn new spatial view features. It effectively solves the problem of template failure caused by scene changes during long-term use and continuously optimizes the accuracy and environmental adaptability of head pose estimation.

[0020] In one possible implementation, the angular velocity cell network comprises a linear continuous attractor network, the spatial view cell network comprises a loop continuous attractor network, and the head orientation cell network comprises a loop continuous attractor network.

[0021] According to another aspect of this disclosure, a head pose estimation device is provided, the device comprising: a data acquisition module for acquiring multimodal data of a target object within a target time period, wherein the multimodal data refers to visual data and inertial data collected by a brain-like visual sensor mounted on the target object; a data preprocessing module for processing the visual data within the target time period along the time direction to obtain a target matrix, and for calculating the joint angular velocity in each direction based on the visual data and inertial data within the target time period; and a first brain-like multicellular interaction module for inputting the joint angular velocity in each direction into an angular velocity cell network to obtain a first pulse sequence representing the change in the joint angular velocity in each direction, and for inputting the target matrix into a spatial... A second pulse sequence representing the pulse firing frequency of head-facing neurons associated with spatial view neurons is obtained in the view cell network, wherein the spatial view neuron refers to a computational unit in the spatial view cell network, and the head-facing neuron refers to a computational unit in the head-facing cell network; a second type of brain multi-cell interaction module is used to input the first pulse sequence into a first intermediary cell network to obtain a first electrical signal representing the input energy of the head-facing neuron, and to input the second pulse sequence into a second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-facing neuron; a fusion calibration module is used to input the first electrical signal and the second electrical signal into the head-facing cell network to obtain the calculation result of the head posture of the target object.

[0022] In one possible implementation, the step of processing the visual data within the target time period along the time direction to obtain the target matrix includes: converting the visual data within the target time period into a first matrix and a second matrix according to preset event types, wherein the event types include brightness increase and brightness decrease, the first matrix represents the event frequency of brightness increase corresponding to each pixel coordinate within the target time period, and the second matrix represents the event frequency of brightness decrease corresponding to each pixel coordinate within the target time period; and performing grayscale conversion based on the first matrix and the second matrix to obtain the target matrix.

[0023] In one possible implementation, the apparatus further includes an image generation module for generating a target image based on the first matrix and the second matrix.

[0024] In one possible implementation, the step of calculating the joint angular velocity in each direction based on visual data and inertial data within the target time period includes: calibrating the inertial data within the target time period to obtain the calibration angular velocity in each direction, and calculating the standard deviation of the angular velocity in the corresponding direction based on the calibration angular velocity in each direction; determining the event angular velocity and pixel matching ratio in each direction based on the visual data within the target time period; and determining the joint angular velocity in the corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity in each direction.

[0025] In one possible implementation, determining the joint angular velocity for a corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity for each direction includes: for each direction, calculating a first ratio based on the angular velocity standard deviation and a preset maximum standard deviation; calculating a first difference based on the preset standard deviation and the first ratio; and using the product of a preset first ratio and the first difference, plus the sum of a preset second ratio, as a first weight for the corresponding direction; calculating a second ratio based on the pixel matching ratio and a preset maximum matching ratio; and using the product of the first ratio and the second ratio, plus the sum of the second ratio, as a second weight; normalizing based on the first weight and the second weight for each direction to obtain a first normalized weight and a second normalized weight for each direction; determining the first angular velocity for the corresponding direction based on the first normalized weight and the calibration angular velocity for each direction, and determining the second angular velocity for the corresponding direction based on the second normalized weight and the event angular velocity; and using the sum of the first angular velocity and the second angular velocity for each direction as the joint angular velocity for the corresponding direction.

[0026] In one possible implementation, the device further includes a verification module, configured to: determine a first motion trend for each target pixel based on visual data within the target time period, and determine a corresponding second motion trend based on the joint angular velocity; if there are a preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be valid; if there are fewer than the preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be invalid, and head pose estimation is performed based on valid multimodal data within historical time periods or by re-acquiring multimodal data within a new target time period.

[0027] In one possible implementation, the device further includes a pose estimation module, configured to: convert the target matrix into a target neuron state matrix according to a preset time encoding rule; calculate the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each preset spatial view template; determine the estimated head pose of the target object based on the head pose calculation result corresponding to the spatial view template with the highest similarity, wherein the estimated result is used to drive attractor state transitions in the spatial view cell network.

[0028] In one possible implementation, the device further includes a template update module, configured to: update the preset spatial view template if the comparison between the calculated head pose determined based on multimodal data within the target time period and the calculated head pose determined based on multimodal data within the previous time period meets a preset condition, and the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset first threshold; and update the preset spatial view template if the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset second threshold.

[0029] In one possible implementation, the angular velocity cell network comprises a linear continuous attractor network, the spatial view cell network comprises a loop continuous attractor network, and the head orientation cell network comprises a loop continuous attractor network.

[0030] According to another aspect of this disclosure, a head pose estimation device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0031] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0032] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0033] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0035] Figure 1 A flowchart illustrating the head pose estimation method provided in an embodiment of this disclosure is shown.

[0036] Figures 2 to 4 A schematic diagram of the head pose estimation method provided in an embodiment of this disclosure is shown.

[0037] Figure 5 A block diagram of a head pose estimation device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0038] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0039] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0040] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0041] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0042] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0043] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0044] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0045] To facilitate understanding of the technical solutions provided by the embodiments of this disclosure by those skilled in the art, the technical environment for implementing the technical solutions will be described below.

[0046] Brain-inspired intelligence is a cutting-edge field of artificial intelligence, focusing on simulating the neural structure and cognitive principles of the brain. Its aim is to enable computing systems to possess human-like perception, reasoning, and learning abilities. Brain-inspired intelligence involves interdisciplinary fields such as brain-inspired computing, multimodal data fusion, and posture perception. In highly dynamic applications such as mobile robots, autonomous aircraft, unmanned racing cars, and virtual reality, accurately acquiring the target's head orientation information is a core prerequisite for achieving autonomous navigation and stable interaction. Currently, the technological evolution in the field of brain-inspired intelligence mainly exhibits the following trends:

[0047] First, there are low-power, adaptive brain-like neural network models. Intelligent computing models are shifting from traditional artificial neural networks to more biologically plausible technologies such as Spiking Neural Networks (SNNs) and Continuous Attractor Neural Networks (CANNs). By simulating the firing patterns of neurons and the information interaction mechanisms between cells, these technologies achieve lower power consumption and higher adaptability in information processing. Currently, this technology has been preliminarily validated in simple scenarios such as low-speed positioning of indoor robots. However, its adaptability and robustness still face significant challenges in high-speed dynamic scenarios with rotation speeds ≥300 revolutions per minute (rpm).

[0048] Second, there is the collaborative fusion of multimodal sensor data. Research is shifting from single-sensor approaches, such as inertial measurement units (IMUs) or vision cameras alone, to multi-sensor collaboration. This involves leveraging the complementary advantages of different sensor modalities (such as the high dynamic response of IMUs and the spatial positioning accuracy of vision cameras) to improve the system's anti-interference performance. Although algorithms such as Kalman filtering, extended Kalman filtering, and fixed-weight fusion have become mainstream, current fusion strategies often lack sufficient adaptability in extreme dynamic scenarios due to the nonlinear changes in the reliability of sensor data.

[0049] Third, high-speed scenarios demand high-precision perception. As the speed of industrial and consumer devices continues to increase, the demand for "high speed, high precision, and high real-time performance" in attitude estimation has also increased significantly. However, related technologies are susceptible to problems such as motion blur and inertial drift under high-speed rotation, which can lead to a significant decrease in perception accuracy or even failure, making it difficult to meet the needs of actual engineering.

[0050] However, these technologies have several drawbacks. Specifically, one is their inability to adapt to high-speed rotating environments. These technologies often use traditional cameras, whose full-frame exposure principle is physically incompatible with high-speed motion. When the rotation speed is ≥300 rpm, the displacement of target feature points within a single exposure cycle far exceeds the pixel size, resulting in severe motion blur. This imaging distortion significantly increases the failure rate of feature point extraction, rendering visual pose measurement results essentially ineffective and unable to provide a valid benchmark for the fusion algorithm. In this situation, the system is forced to degenerate into a single inertial navigation mode, leading to a significant decrease in measurement accuracy.

[0051] Secondly, the system suffers from weak anti-interference capabilities and insufficient information processing refinement. The current solution employs a unidirectional linear process of "inertial calculation - visual calculation - filtering fusion," where multimodal data is simply weighted in the final fusion stage without deep correlation. This loosely coupled architecture results in weak anti-interference capabilities. When visual information fails during high-speed motion, the system lacks effective alternative information compensation, easily causing interruptions or jumps in attitude head orientation output, thus failing to maintain continuous perception.

[0052] Third, the accuracy of spatial positioning and inertial data is insufficient. The current solution only calculates the relative angular rate through IMU differential calculation. However, at high speeds, centrifugal force causes the IMU accelerometer measurement to deviate from the true direction of gravity, while also exacerbating the dynamic drift of the gyroscope. Relying solely on IMU calibration is insufficient to completely offset the compound error caused by centrifugal force, resulting in a decrease in the accuracy of angular velocity calculation. Furthermore, after integration, it amplifies the error in attitude and head orientation calculation, ultimately leading to unreliable head orientation judgment results.

[0053] Fourth, fixed fusion strategies struggle to cope with dynamic changes in multimodal information. Current solutions employ fixed-parameter extended Kalman filter fusion, which cannot handle the nonlinear changes in modal reliability at high speeds. When visual motion blur fails at high speeds or IMU drift intensifies due to centrifugal force, fixed parameters still allocate weights according to low-speed logic, easily leading to the system being misled by low-reliability observation data, resulting in decreased fusion accuracy or even failure.

[0054] In summary, the overall design of the related technologies did not take into account the extreme dynamic characteristics of high-speed rotating scenarios. The hardware selection, fusion strategy, and information processing architecture were all based on low-speed motion scenarios, resulting in problems such as visual modality failure, a sharp drop in fusion accuracy, and poor accuracy and stability of head orientation estimation at high speeds. As a result, it is difficult to meet the needs of core application scenarios such as robots, drones, unmanned racing cars, and virtual reality in high-speed scenarios.

[0055] This disclosure provides a head pose estimation method for high-speed rotating scenarios. It employs a brain-like multi-cell interaction architecture and a brain-like visual sensor-IMU multimodal deep fusion technology to address core shortcomings of related technologies in high-speed rotating scenarios, such as visual imaging failure and rigid fusion strategies. This method addresses four key technical issues through targeted technical design, ultimately achieving high accuracy, high real-time performance, and high stability in head orientation determination in high-speed scenarios to meet the needs of core application scenarios such as robotics, drones, unmanned racing cars, and virtual reality.

[0056] Firstly, this method addresses the issue that relying solely on IMU differential calculation of angular velocity and centrifugal force at high speeds exacerbates angular velocity accuracy problems caused by IMU drift. This method constructs an IMU-event optical flow joint angular velocity calculation mechanism, deeply coupling the dynamic response advantages of the IMU with the anti-motion blurring advantages of the brain-like visual sensor, thereby improving the accuracy of angular velocity calculation and anti-interference capability at high speeds.

[0057] Secondly, this method addresses the issues of motion blur visual positioning failure and uncorrectable IMU drift in traditional cameras at high speeds. It introduces grayscale frame template matching technology, which generates grayscale frames based on the asynchronous event stream output by the neuromorphic vision sensor and accurately matches them with the pre-stored spatial view template. This provides reliable head orientation positioning results at high speeds and corrects accumulated IMU drift in real time, avoiding unconstrained growth of head orientation estimation errors caused by visual modality failure.

[0058] Third, to address the problem that related technologies employ linear information processing flows and lack the ability to simply weight and fuse multimodal data or to achieve refined integration, this method designs a brain-like multicellular interaction architecture. By simulating the division of labor and collaboration among angular velocity cells (AVC), spatial view cells (SVC), head direction cells (HDC), angular velocity-head direction mediator networks, and spatial view-head direction mediator networks in biological navigation systems, this method achieves specialized processing and deep coupling of motion information and spatial visual information. This avoids the interruption of posture and head direction information caused by the failure of a single modality, thus ensuring the continuity of perception.

[0059] Fourth, to address the issues of rigid integration strategies for related technologies, inability to adapt to dynamic changes in modal reliability at high speeds, and difficulty in balancing accuracy and real-time performance, this method establishes an adaptive information fusion mechanism by integrating three core methods: IMU-event information joint angular velocity calculation, event template matching, and brain-like multicellular interaction. This ensures a balance between accuracy, real-time performance, and stability in head orientation judgment during high-speed rotation scenarios.

[0060] Now combined Figures 1 to 4 The head pose estimation method provided in this disclosure is illustrated. This method can be applied to high-speed head orientation estimation and control scenarios such as robots, drones, unmanned racing cars, and virtual reality. Figure 1 As shown, the head pose estimation method may include the following steps S101 to S105.

[0061] Step S101: Obtain multimodal data of the target object within the target time period. Multimodal data refers to visual data and inertial data collected by a brain-like visual sensor set on the target object.

[0062] To acquire multimodal data, the neuromorphic vision sensor includes not only a dynamic vision sensor but also an inertial measurement unit (IMU). Therefore, the multimodal data includes visual data from the dynamic vision sensor and high-frequency inertial data from the IMU (refer to...) Figure 2 The visual data output by the dynamic vision sensor may include timestamps, pixel coordinates (x, y), and event polarity. Event polarity / type includes brightness increase and brightness decrease, with brightness increase recorded as +1 and brightness decrease as -1. The inertial data output by the inertial measurement unit may include the angular velocity of the target object in each direction (x-axis, y-axis, z-axis). For example, the target time period or time window may be preset to 2ms. Multimodal data may include (x, y, polarity) at each timestamp within 2ms and the angular velocities of the target object around the x-axis, y-axis, and z-axis within 2ms. In addition to dynamic vision sensors or event cameras, other sensors such as pulse cameras and neuromorphic complementary vision cameras can also be used, and this disclosure does not limit the types of sensors used.

[0063] Step S102: Process the visual data within the target time period along the time direction to obtain the target matrix, and calculate the joint angular velocity in each direction based on the visual data and inertial data within the target time period.

[0064] The step S102, which processes the visual data within the target time period along the time direction to obtain the target matrix, may include: converting the visual data within the target time period into a first matrix and a second matrix according to preset event types, wherein the event polarity / type includes brightness increase and brightness decrease, the first matrix represents the event frequency of brightness increase corresponding to each pixel coordinate within the target time period, and the second matrix represents the event frequency of brightness decrease corresponding to each pixel coordinate within the target time period; and performing grayscale conversion based on the first matrix and the second matrix to obtain the target matrix.

[0065] This method employs a dynamic time window accumulation strategy to transform the visual data (i.e., asynchronous event streams / sequences) output by a neuromorphic vision sensor into a structured tensor [timestamp, x, y, polarity] with spatial topological features. For example, firstly, time-slice accumulation is performed. Specifically, according to a preset time window, the visual data is sliced ​​along the time axis, and the intensity is accumulated according to the event polarity and projected onto a two-dimensional pixel plane. For instance, in the two-dimensional pixel plane corresponding to the event polarity of brightness increase (brightness rise), the frequency of brightness rise events occurring at each pixel coordinate point within that window is accumulated to obtain the corresponding two-dimensional pixel count matrix, i.e., the first matrix. The transformation process of the second matrix is ​​similar to that of the first matrix and will not be elaborated further. Then, standard grayscale processing is performed. Specifically, a maximum value normalization algorithm can be applied to map the response intensities of the first and second matrices to the grayscale range of [0, 255], generating a standardized grayscale frame, i.e., the target matrix, for subsequent template matching. In this example, the default time window is 2ms, but the time window is actually adjustable within the range of 1-5ms, and can be flexibly adjusted according to the actual situation. Normalization can be performed using the following formula:

[0066]

[0067] In the formula, This represents the matrix after grayscale conversion. This refers to the cumulative frequency of events at the corresponding pixel coordinates (x, y). This indicates the maximum cumulative frequency of events.

[0068] In this way, the method converts asynchronous event streams into structured data that meets the requirements of neuromorphic neural computing, while retaining the advantages of neuromorphic visual sensors such as microsecond-level response and anti-motion blur, and avoiding the blurring problem caused by full-frame exposure at high speeds in traditional RGB cameras.

[0069] The head pose estimation method may also include: generating a target image based on a first matrix and a second matrix. The first matrix corresponds to the brightness up channel, and the second matrix corresponds to the brightness down channel. A high-contrast, visualized target image / frame is generated through dual-channel color mapping (brightness up is marked in red, brightness down is marked in blue).

[0070] Step S102, which calculates the joint angular velocity in each direction based on visual and inertial data within the target time period, may include: calibrating the inertial data within the target time period to obtain the calibrated angular velocity in each direction, and calculating the standard deviation of the angular velocity in the corresponding direction based on the calibrated angular velocity in each direction; determining the event angular velocity and pixel matching ratio in each direction based on the visual data within the target time period; and determining the joint angular velocity in the corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibrated angular velocity in each direction. This method employs a deep coupling strategy between neuromorphic vision and inertial data, proposes a joint angular velocity calculation mechanism combining IMU and event information, and combines it with a spatial view template matching technique based on VPRTempo using pulse time coding to spatiotemporally align motion information with spatial view features, significantly improving the system's perception input accuracy in high-speed rotating scenes.

[0071] For example, for calculating the calibration angular velocity, 30s of static reference data (i.e., the angular velocities of the target object around the x, y, and z axes when it is stationary) can be collected, and the average value in each direction can be taken as the zero bias. Based on the zero bias, the original angular velocity can be calculated. Real-time de-biasing compensation. Taking the x-axis as an example, the angular velocity along the x-axis is obtained from the 2ms inertial data collected. Subtracting the zero bias yields the calibrated angular velocity along the x-axis. Angular velocities of the y and z axes. , The calculation process is similar to Further details are omitted. The 30s setting is merely an example; the actual duration can be adjusted based on specific circumstances.

[0072] For example, the standard deviation of angular velocity can be calculated based on the calibrated angular velocity. Taking the x-axis as an example, the standard deviation of the angular velocity along the x-axis is calculated using the calibrated angular velocity over a period of 2 ms after calibration. The standard deviation of the angular velocity along the y-axis and z-axis is similarly... I won't go into details. The smaller the standard deviation of angular velocity, the higher the inertial stability.

[0073] For example, the event angular velocity can be calculated based on event optical flow. Specifically, the displacement of pixel coordinates in the event frames (i.e., the target image) corresponding to adjacent time windows can be calculated based on camera intrinsic parameters and the projection geometry of the pixel plane, and then the motion velocity can be calculated by combining the time window length. Finally, by combining the preset installation radius r, the event angular velocities corresponding to the x and y coordinate axes are calculated in reverse. In this example, there is no lateral movement along the z-axis by default, but the specific setting should be adjusted according to the actual situation.

[0074] For example, the calculation of the pixel matching ratio can be performed by comparing the feature pixel matching ratios in the overlapping regions of the event frames (i.e., the target image) within adjacent time windows after the event angular velocity rotation. .for example, ≥80% indicates that the visual data is reliable. 80% is a preset threshold and can be set according to actual conditions.

[0075] The determination of the joint angular velocity for each direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity can include: for each direction, calculating a first ratio based on the angular velocity standard deviation and a preset maximum standard deviation; calculating a first difference based on the preset standard deviation and the first ratio; and using the product of the preset first ratio and the first difference, plus the sum of the preset second ratio, as the first weight for the corresponding direction; calculating a second ratio based on the pixel matching ratio and a preset maximum matching ratio; and using the product of the first ratio and the second ratio, plus the sum of the second ratio, as the second weight; normalizing based on the first and second weights for each direction to obtain a first normalized weight and a second normalized weight for each direction; determining the first angular velocity for the corresponding direction based on the first normalized weight and the calibration angular velocity, and determining the second angular velocity for the corresponding direction based on the second normalized weight and the event angular velocity; and using the sum of the first and second angular velocities for each direction as the joint angular velocity for the corresponding direction.

[0076] For example, taking the x-axis as an example, the combined angular velocity can be calculated according to the following formula:

[0077]

[0078] In the formula, Represents the combined angular velocity along the x-axis. This represents the first normalized weight along the x-axis. Indicates the calibrated angular velocity along the x-axis. Represents the first angular velocity along the x-axis. This represents the second normalized weight. Represents the event angular velocity along the x-axis. This represents the second angular velocity along the x-axis.

[0079] The first and second normalized weights along the x-axis can be calculated using the following formula:

[0080]

[0081]

[0082]

[0083] In the formula, This represents the first weight along the x-axis. This represents the standard deviation of the angular velocity along the x-axis. This represents the preset maximum standard deviation. Indicates the first ratio. Indicates the first difference. Indicates the first proportion. Indicates the second proportion. Indicates the second weight. This indicates the pixel matching ratio; 100 represents the preset maximum matching ratio. This represents the second ratio. This represents the first normalized weight along the x-axis. This represents the second normalized weight. In this example, It can be set to 0.5. It can be set to 0.3. It can be set to 5° / s, but these parameters can be set according to the actual situation.

[0084] In this example, it is assumed that there is no lateral movement along the z-axis, and only the joint angular velocity in the x and y directions is calculated. However, the directions involved in solving the joint angular velocity depend on the actual device settings, and this embodiment does not limit them.

[0085] Step S102 enables data preprocessing of multimodal data. Thus, this method leverages the stability advantage of the IMU to compensate for visual processing latency, while utilizing the anti-blurring characteristics of event optical flow to calibrate the centrifugal force deviation of inertial devices under high-speed rotation, significantly improving the accuracy of angular velocity calculation and anti-interference capability in high-speed rotating scenes.

[0086] The head pose estimation method may further include: determining a first motion trend for each target pixel based on visual data within a target time period, and determining a corresponding second motion trend based on joint angular velocity; if there are more than or equal to a preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be valid; if there are less than a preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be invalid, and head pose estimation is performed based on valid multimodal data within historical time periods or by acquiring new multimodal data within the target time period. Here, the target pixels are derived from the target image.

[0087] In head pose estimation methods, such as Figure 3As shown, after acquiring the multimodal data output by the neuromorphic vision sensor, data consistency verification can be performed to ensure that the multimodal data currently used for head pose estimation is valid. Specifically, the motion trend can be logically verified by comparing the integral trend of the joint angular velocity with the spatial changes of feature points within the event frame (i.e., the target image). The first motion trend of each target pixel / feature point is determined based on the visual data within the target time period, and the second motion trend is determined based on the joint angular velocity. The first motion trend is determined based on visual information, while the second motion trend is determined based on IMU data; their judgment criteria differ. A consistency threshold of 0.8 can be set, so that when the displacement trend of more than 80% of the feature points (i.e., the first motion trend) matches the angular velocity direction (i.e., the second motion trend), the acquired multimodal data within the current target time period is considered valid. If less than 80% of the feature points have the same first motion trend as the second motion trend (i.e., the consistency coefficient is lower than the preset consistency threshold of 0.8), then the multimodal data obtained in the current target time period is determined to be abnormal jitter or sensor lock-off, and then resampling (event stream and IMU data in multiple time frames) or smooth prediction, i.e. head pose estimation, is triggered using effective multimodal data in the nearby historical time period.

[0088] In this way, the method effectively filters out abnormal disturbance data through logical verification, ensures the semantic and logical consistency of the input multimodal data, and avoids system output interruption caused by single modality failure.

[0089] Head pose estimation methods also involve high-speed scene spatial view template matching based on temporal coding. To this end, this method uses pulse temporal coding techniques / rules, utilizes neuronal state matrices to represent spatial features, and achieves accurate matching between the high dynamic contours captured by a brain-like vision sensor and pre-stored high-speed scene spatial view templates, thus solving the matching failure problem of existing RGB image templates caused by image degradation at high speeds.

[0090] Specifically, the head pose estimation method may further include: converting the target matrix into a target neuron state matrix according to a preset time encoding rule; calculating the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each preset spatial view template; determining the estimated head pose of the target object based on the head pose calculation result corresponding to the spatial view template with the highest similarity, and the estimated result is used to drive the attractor state transition in the spatial view cell network.

[0091] For a pre-defined spatial view template, a neuromorphic spatial view template library can be pre-built to achieve robust representation of specific scenes (such as track textures, laboratory markers, and VR fixed reference objects). Specifically, for a given scene, data output from a brain-like visual sensor, such as a 2ms event stream, can be acquired multiple times. Based on this data, a template matrix for that scene can be obtained, which represents the spatial view template. The head pose estimation method can also include converting the template matrix into a template neuron state matrix according to a pre-defined temporal encoding rule.

[0092] For example, the template neuron state matrix corresponding to the spatial view template can be represented using a time-coding format based on neural impulses (i.e., a preset time-coding rule): First, impulse coding mapping can transform the raw event stream in high-speed scenes (such as 2ms visual data) into a template neuron state matrix. This matrix includes three dimensions: camera resolution, time step, and impulse features. The brightness change frequency of pixels is mapped to impulse amplitude (normalized to the [0,1] interval), while the relative time of event occurrence is mapped to the impulse triggering sequence. Second, multi-dimensional data encapsulation: the dimensions of the template neuron state matrix can be defined as camera resolution × discrete time step × impulse features (the time step is 2ms by default, consistent with the time window). Each spatial view template encapsulates the corresponding spatial label, applicable motion speed range (300rpm-1500rpm), absolute head orientation, and impulse timing annotations for key feature points, ensuring complete alignment with the encoding logic and data format of the front-end event frames (i.e., the real-time acquired visual data).

[0093] The target matrix (i.e., grayscale frames) is converted into a target neuron state matrix according to the aforementioned time-coding rules. Specifically, the pixel intensity indicated by the target matrix is ​​first mapped to neuron pulse amplitude. Pixel grayscale values ​​(0-255) are proportionally normalized to the 0-1 range; higher grayscale values ​​correspond to pulse amplitudes closer to 1. Simultaneously, based on a time window, the state of the brain is abstracted. The oscillation time step and the timing information of pixel brightness changes directly correspond to the triggering timing of the pulse within that time step. Subsequently, a neuron state matrix is ​​constructed by combining the image resolution and the time step. The matrix dimension matches the event camera resolution and the set time step. Each matrix element simultaneously carries the pulse amplitude and triggering timing information of the corresponding pixel, ultimately forming structured data consistent with the pre-stored spatial view template encoding format.

[0094] To calculate the similarity between the target neuron state matrix and the template neuron state matrix, a normalized cross-correlation extension algorithm can be used to compare the neuron states. This algorithm compares both the pulse intensity in the pixel dimension and the pulse firing order in the time dimension. The logical formula for neuron state similarity is as follows:

[0095]

[0096] In the formula, each parameter corresponds to the spatiotemporal coding features of the three-dimensional neuron state matrix: Represents the pixel space coordinates of the event camera, corresponding to the spatial dimension of the neuron state matrix; Represents a discrete time step, corresponding to the time dimension of the matrix; Indicates the current preprocessed event frame in ( The real-time neuron state value at the location, i.e., the pulse amplitude; The weight coefficients of the pre-built template at the corresponding spatiotemporal location are used to highlight the pulse features of key feature points in the scene and weaken redundant backgrounds; This indicates that the corresponding spatial view template in the neuromorphic spatial view template library is located in ( Standard neuron state values ​​at the location; This represents the global summation of all spatial pixels and time steps traversing the entire 3D neuron state matrix; the final output is... The value represents the spatiotemporal similarity in the range of 0 to 1. The closer the value is to 1, the higher the matching degree between the input target neuron state matrix and the template neuron state matrix corresponding to the spatial view template.

[0097] The absolute orientation estimate, i.e., the estimated head orientation of the target object, is output based on the head pose calculation result corresponding to the spatial view template with the highest similarity. Specifically, when the similarity meets the judgment threshold (i.e., similarity ≥ 0.7), the preliminary estimate of the head orientation is calculated in real time based on the head pose calculation result corresponding to the highest similarity and the spatial view template with the highest similarity (i.e., the head orientation corresponding to the template), thus achieving absolute positioning in a highly dynamic environment.

[0098] The head pose estimation method also involves the dynamic updating of the spatial view template. To cope with the nonlinear evolution of the environment, this method also establishes a dynamic maintenance / update mechanism that requires no manual intervention. Specifically, the head pose estimation method may further include: if the comparison between the calculated head pose (including head orientation) determined by multimodal data within the target time period and the calculated head pose determined by multimodal data within the previous time period meets preset conditions, and the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset first threshold, then the preset spatial view template is updated. For example, if the difference between the current head orientation and the previous head orientation is greater than 10° (e.g., x-axis direction change > 10°) and the similarity between the current target neuron state matrix and each spatial view template is less than the first threshold, such as 0.4, then a forced update logic can be triggered to prevent non-physical errors in the localization results; if the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset second threshold, then the preset spatial view template is updated. For example, if the similarity between the current target neuron state matrix and the state matrix of all spatial view template samples in the template library is lower than the second threshold, such as 0.5, then it can be identified as a new scene feature, and the target neuron state matrix is ​​automatically encoded into the library. In addition, if the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset third threshold in multiple consecutive target time periods, the preset spatial view template will be updated. For example, if the similarity between the target neuron state matrix and the template neuron state matrix in five consecutive time windows is lower than the third threshold, such as 0.6, the current dominant template will be determined to be invalid and the spatial view template will be updated.

[0099] This paper presents a head pose estimation method based on a pulsed continuous attractor network (STDP) model. By simulating the multi-cellular division of labor and interaction mechanisms of biological nervous systems, it achieves deep integration and dynamic calibration of multimodal information. Addressing the shortcomings of rigid fusion strategies and coarse information processing in related technologies, this method employs a continuous attractor neural network combined with a leakage integral ignition model, while introducing the pulsed time-dependent plasticity mechanism to optimize the synaptic weights across the entire link in real time. This architecture ensures continuous and stable head pose encoding even under high-speed rotation.

[0100] The head orientation estimation model in this case adopts a three-layer network architecture, namely the input layer, the intermediate layer and the output layer.

[0101] The input layer is a continuous attractor network driven by multimodal input. The angular velocity cell network and the spatial view cell network serve as the perceptual input layer, and perform morphological adaptation encoding for the physical characteristics of angular velocity and spatial view, respectively.

[0102] Step S103: Input the joint angular velocity of each direction into the angular velocity cell network to obtain a first pulse sequence representing the change of the joint angular velocity in each direction; and input the target matrix into the spatial view cell network to obtain a second pulse sequence representing the pulse firing frequency of the head-facing neurons associated with the spatial view neurons. Here, the spatial view neuron refers to the computational unit in the spatial view cell network, and the head-facing neuron refers to the computational unit in the head-facing cell network.

[0103] Angular velocity cell networks are linear continuous attractor networks adapted to one-dimensional motion spaces. Their topology employs a linearly arranged array of neurons (e.g., ...). Figure 4 As shown), it covers a continuous angular velocity space from -50° / s to 50° / s (flexibly configurable). Neurons form linear attractors through local excitation connections, supporting smooth state transitions. The internal synaptic weights follow a linear Gaussian distribution, and the connection strength is dynamically fine-tuned based on the temporal difference between pre- and post-synaptic pulses using a pulse time-dependent plasticity mechanism. Under the constraint of global inhibitory current, a single active center is maintained, avoiding discrete state transitions. Its input is the joint angular velocity of the IMU-event optical flow, which directly drives the linear displacement of the attractor state. The firing frequency of neurons in the activated region is positively correlated with the angular velocity magnitude, outputting a presynaptic pulse sequence carrying motion information downstream. (i.e., the first pulse sequence). This indicates the change in the combined angular velocity in each direction.

[0104] The spatial view cell network is a loop-shaped continuous attractor network, adapted to the cyclicity of the spatial perspective, with neurons arranged in a closed loop in the topology (e.g., Figure 4 As shown), it maps a cyclic space from 0° to 360°. The ring-shaped connection characteristic ensures the seamless connection of spatial perspectives (such as the neighborhood relationship between 359° and 1°). The initial weights inside have a ring-shaped Gaussian distribution. Through the pulse time-dependent plasticity mechanism, it learns in real time to compensate for dynamic environmental interference. Under the global suppression current regulation, it maintains a single continuous activation region, accurately corresponding to the current spatial visual absolute head orientation. Its input is a grayscale frame, i.e., the target matrix. The attractor state cyclical migration is driven by the preliminary estimate of the head orientation matched by the spatial view template. The cell pulse firing frequency in the activation region is positively correlated with the matching similarity. The presynaptic pulse sequence is output downstream. (i.e., the second pulse sequence). This indicates the pulse firing rate of the head-oriented cell / neuron corresponding to the spatial view cell / neuron.

[0105] The intermediate layer is an intermediary network, which includes the angular velocity cell-head orientation cell intermediary cell (AVC-HDC) network (i.e., the first intermediary cell network) and the spatial view cell-head orientation cell intermediary cell (SVC-HDC) network (i.e., the second intermediary cell network). Its core function is to realize the nonlinear mapping from the motion parameter space to the absolute orientation space and complete the cross-modal signal connection.

[0106] Step S104: Input the first pulse sequence into the first intermediary cell network to obtain a first electrical signal representing the input energy of the head-oriented neuron, and input the second pulse sequence into the second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-oriented neuron.

[0107] Angular velocity cell-head orientation cell intermediate cell network, i.e., the first intermediate cell network, such as... Figure 4 As shown, a linear-to-circular transitional pattern is adopted as an interactive bridge between motion increment and orientation state. Its cell arrangement exhibits a geometrical gradient from linear to circular. Through synaptic weights optimized by the pulse time-dependent plasticity mechanism, the linear displacement of the angular velocity cell network is converted into an energy injection component of the head-to-cell network circular space. This ensures that the head orientation encoding achieves a smooth cycle as the angular velocity is integrated. It is transmitted between cells in the form of pulse trains. This is the first electrical signal.

[0108] Spatial view of cell-head orientation of the intermediate cell network, i.e., the second intermediate cell network, such as... Figure 4 As shown, a ring-to-ring adapter type is adopted, and a ring topology is used. The inner ring aligns with the spatial view of the spatial view cell network, and the outer ring aligns with the head orientation of the cell network. By maintaining the phase synchronization of the dual-ring attractors, the visual positioning signal is directly mapped to the absolute orientation calibration component of the head orientation cell network. This corrects drift during the integration process. It is transmitted between cells in the form of pulse trains. This is the second electrical signal.

[0109] The output layer, also known as the state integration layer, is a head-oriented, cell-loop continuous attractor network (referred to as head-oriented cell network). In this method, all subnetworks in the input, intermediate, and output layers adopt continuous attractor neural networks (CANN) and are combined with a leakage integral ignition (LIF) model to construct a multimodal information continuous integration system that simulates biological neural circuits, which is physically adapted to the real-time and cyclic evolution requirements of high-speed motion data.

[0110] Step S105: Input the first electrical signal and the second electrical signal into the head orientation cell network to obtain the calculation result of the head posture of the target object.

[0111] The head orientation cell network is a continuous loop attractor network. It serves as the integration center for the system's final head orientation state, and its state update relies on the dynamic evolution of the loop attractors. The head orientation cell network receives energy components injected by two types of intermediate networks in the middle layer—the first electrical signal and the second electrical signal. Through the cooperative evolution of the loop attractors, it achieves deep fusion of multimodal information. Finally, the head orientation angle is encoded by the central position of the activated region of the loop attractor, outputting a precise and continuous final head orientation result, i.e., the calculated head posture. The dynamic evolution of the attractors may involve membrane potential dynamic equations and adaptive firing rate and global suppression.

[0112] Regarding the membrane potential dynamics equation, the neuronal membrane potential is jointly driven by the internal recurrent excitatory connections and the energy injection from the multi-path mediating network. A weight matrix regulated by a pulse-time-dependent plasticity mechanism ensures high physical consistency in the evolution of the attractor within a 360° radius, seamlessly handling boundary continuity issues across the 0° point and preventing coding breaks in high-speed scenarios. The cell membrane potential can be expressed as follows:

[0113]

[0114] In the above formula Indicates the spatial location on the ring structure of the HDC network. Neurons at the location Membrane potential at time t, Indicates the spatial location on the ring structure of the HDC network. Neurons at the location Membrane potential at time t; The membrane potential decay time constant controls the decay rate of the membrane potential. It is the global excitation gain coefficient, used to adjust the excitation conduction strength of intercellular connections; For the location in the ring network There are spatial locations of connected adjacent neurons; It is a location and The excitation connection weights between neurons initially exhibit a circular Gaussian distribution, with higher weights for cells at adjacent angles. This is dynamically adjusted via STDP to maintain the closed, continuous structure of the circular attractor. It is a location Neuron at The pulse firing status at any given moment; , These are two types of intermediary networks directed to location. Energy signals injected into neurons for Time and location The firing rate of neurons is an adaptive variable responsible for regulating global inhibition.

[0115] Regarding adaptive firing rate and global suppression, adaptive firing rate variable Both the global suppression weights and the HDC network are adjusted through a pulse-time-dependent plasticity mechanism. This allows the HDC network to suppress non-target background noise and maintain a single, sharp activation peak even when there is a mismatch between the conflict angular velocity calculated from multimodal data and the visual template output, thus locking in a unique head orientation estimate. The update of the firing rate adaptive variable can be expressed as follows:

[0116]

[0117] In the formula, express Time and location The firing rate of neurons is an adaptive variable. This represents the time constant for adaptive disbursement rates. This represents the adaptive gain coefficient. The other parameters are described above and will not be repeated here.

[0118] The activated cells continuously fire pulses, and the corresponding annular spatial position represents the continuous attractor state of the current head orientation. Neuron at The pulse firing state at a given time can be expressed as follows:

[0119]

[0120] In the formula, Indicates position Neuron at The pulse firing status at any given moment. This represents the membrane potential threshold at which the neuron fires the trigger pulse. Other parameters are described above and will not be repeated here.

[0121] The method provides a full-link weight adjustment mechanism based on pulse time-dependent plasticity, which introduces the pulse time-dependent plasticity (STDP) rule mechanism to dynamically adjust the excitatory connection weights and global inhibition weights within the network and across modules online. This enables deep fusion of multimodal data at the neuron level and ensures the stability of attractor states in dynamic environments.

[0122] This method employs a multimodal fusion logic based on the cooperative evolution of full-link attractors. A complete hierarchical continuous attractor system is constructed from angular velocity cell networks, spatial view cell networks, and intermediate networks. Its fusion logic is manifested as follows: a dynamic tracking chain, where, at high speeds, the angular velocity cell network-driven "linear-transition-loop" dynamic link achieves incremental head-orientation tracking; and a closed-loop calibration chain, where the spatial view cell network-driven "dual-loop synchronization" link provides an absolute head-orientation reference, performing closed-loop forced correction of drift generated by the tracking chain.

[0123] In this way, the end-to-end continuous attractor ensures lossless integration of multimodal information in the spatiotemporal dimension. Combined with the adaptive weight adjustment of the pulse time-dependent plasticity mechanism, high precision, high continuity, and high robustness of head orientation encoding under high-speed rotation environment are finally achieved.

[0124] This method proposes a differentiated continuous attractor morphology adaptation design. Customized attractor topologies, including linear, loop, and transitional types, are designed to address the linear distribution of angular velocity, the cyclical characteristics of spatial viewpoints, and the orientational characteristics of head orientation. Combined with nonlinear mapping of the mediator network, this achieves smooth transitions and precise mapping of multimodal states. Furthermore, this method proposes a configurable brain-like multicellular interaction architecture, supporting global configuration of neuron size and core parameters (such as angular velocity threshold and viewpoint coverage) for angular velocity cells, spatial view cells, head orientation cells, and mediator network cells. This highly flexible architecture allows for rapid adaptation to the needs of different platforms and tasks.

[0125] This method systematically addresses the core shortcomings of related technologies in ultra-high-speed scenarios, such as perception failure, policy rigidity, and accuracy drift, through a core technical solution that incorporates a full-link continuous attractor network, adaptive weight adjustment based on pulse time-dependent plasticity mechanism, and deep multimodal fusion of a brain-like visual sensor-IMU. Compared to related technologies, this method achieves several technological breakthroughs and possesses significant industry application value.

[0126] First, this method overcomes the bottleneck of high-speed perception, ensuring continuous perception in highly dynamic environments. Utilizing the asynchronous triggering and frame-free exposure delay characteristics of neuromorphic vision sensors, it completely solves the feature failure problem caused by motion blur in traditional RGB cameras during high-speed rotation (≥300 rpm). By leveraging the smooth state transition characteristics of the end-to-end continuous attractor network, head orientation encoding maintains a continuous trajectory even under high-speed rotation, eliminating signal interruptions and output jumps common in high-speed scenarios and filling a technological gap in high-speed directional detection.

[0127] Second, the method simultaneously improves the accuracy and anti-interference capability of multimodal fusion. It offers the advantage of optimized data accuracy by utilizing a joint computational mechanism of IMU-brain-like visual sensor optical flow. Leveraging the complementarity of the two types of sensors in the temporal and spatial domains, and combined with refined coding of the angular velocity cell network, it effectively suppresses IMU axial drift caused by high-speed centrifugal force. Furthermore, by incorporating pulse-time coding technology, it enhances the determinism of spatial positioning references. This method also boasts enhanced robustness. The single-activation region characteristics of the continuous attractor network, combined with the pulse-time-dependent plasticity mechanism, can spontaneously filter random noise generated by high-frequency vibrations and the electromagnetic environment. Even when a single sensor mode experiences instantaneous physical failure, the intermediate network can still maintain the system state through attractor energy injection, ensuring the operational stability of robots or drones under complex conditions.

[0128] Third, the fusion strategy shifts from a static weighting to a dynamic adaptive paradigm. This method introduces a pulse-time-dependent plasticity mechanism, achieving nonlinear decoupling between the fusion logic and modal reliability. It can perceive quality fluctuations in sensor data in real time. When inertial data is stable, it automatically enhances the synaptic strength between angular velocity cells and head-orientation cells; when visual matching is dominant, it strengthens the calibration weights of spatial view cells on head-orientation cells. This self-learning capability overcomes the drawback of traditional fixed-parameter filters where low-quality data dominates fusion under extreme dynamic conditions, significantly improving the system's environmental adaptability.

[0129] Fourth, the method optimizes computational energy efficiency to meet the real-time and long-endurance requirements of edge devices. This method boasts high real-time response; the neuromorphic multi-cell interaction architecture, through parallelized processing, significantly reduces the computational latency from perception to state update, greatly improving data processing efficiency and meeting the demands for real-time attitude and head-on control. Furthermore, this method offers low power consumption; the core algorithm is highly compatible with the asynchronous perception characteristics of neuromorphic vision sensors, allowing deployment on low-power neuromorphic chips or general-purpose embedded platforms. This significantly reduces the system's peak power consumption, effectively alleviating the stringent limitations on battery life imposed by drones and VR wearable devices.

[0130] Fifth, the system architecture possesses extremely high fault tolerance and parameterized scalability. This method ensures robustness; the end-to-end continuous attractor network exhibits disturbance resistance and self-stabilization characteristics. Even if some neuron nodes fail, the system can still maintain the attractor state through the excitation connections of adjacent cells. Simultaneously, weight adjustment utilizes a pulse-time-dependent plasticity mechanism to adapt to scene changes online without retraining the model, making it suitable for flexible applications of robots in complex working environments and virtual reality in diverse scenarios. Furthermore, it adapts to multiple scenarios; the cell size and range of angular velocity cells, spatial view cells, head-oriented cells, and mediator cells can all be customized according to the actual platform (such as micro-drones, large industrial robotic arms, etc.), demonstrating strong engineering scalability.

[0131] This method provides highly adaptable technical support for the core needs of cutting-edge fields such as industrial robots, autonomous aerial vehicles, unmanned racing cars, and virtual reality. Specifically, it ensures the precision of end effector attitude control and system robustness during high-speed rotating grasping or complex trajectory operations, effectively avoiding positioning deviations caused by motion ambiguity. In high-speed flight or high-dynamic racing environments, it provides a fast-response, highly interference-resistant head-pointing reference, meeting the stringent requirements of control closed-loop for high-real-time orientation data. In virtual reality devices, it enhances the immersive experience through low-latency, highly stable head-pointing judgment. Furthermore, this method utilizes mature commercial components, requires no special customized hardware, and offers flexible deployment of the core algorithm, possessing significant potential for large-scale application and driving the upgrade and iteration of high-speed attitude head-pointing detection technology.

[0132] The high-speed head orientation determination method, i.e., head posture estimation method, driven by a full-link continuous attractor network provided in this disclosure realizes head orientation calculation through angular velocity cell networks, spatial view cell networks, angular velocity cell-head orientation cell intermediary cell networks, spatial view cell-head orientation cell intermediary cell networks, and HDC head orientation cell networks. These networks adopt linear, cyclic, linear-cyclic transition topologies, double-cyclic nested adaptation topologies, and cyclic topologies, respectively, to jointly realize the continuous integration and state representation of multimodal motion vectors and absolute head orientation information. This method adopts a dynamic adjustment mechanism of neuronal weights based on pulse time-dependent plasticity (STDP) and applies it to the above-mentioned continuous attractor system. By detecting the firing timing correlation of pulses between synapses, and according to the pulse time-dependent plasticity (STDP) rule mechanism, it dynamically corrects the excitatory connection weights and global inhibition coefficients within each sub-network and across modules; realizes the nonlinear autonomous allocation of multimodal fusion weights, and ensures the evolutionary stability of attractor states in dynamic environments. This method also allows for independent adjustment of core technical parameters such as the number of neurons, sensing range (e.g., maximum angular velocity threshold), and spatial view coverage of each network. This enables the system to adapt to different robot platforms or mobile operation scenarios through parameter mapping without changing the underlying algorithm architecture. Furthermore, this method provides highly accurate raw perceptual inputs to the angular velocity cell (AVC) network and the spatial view cell (SVC) network by spatiotemporally aligning high-frequency inertial data with motion-blurred asynchronous visual features.

[0133] This disclosure also provides a head pose estimation device, comprising: a data acquisition module for acquiring multimodal data of a target object within a target time period, wherein the multimodal data refers to visual data and inertial data collected by a brain-like visual sensor mounted on the target object; a data preprocessing module for processing the visual data within the target time period along the time direction to obtain a target matrix, and for calculating the joint angular velocity in each direction based on the visual data and inertial data within the target time period; and a first brain-like multicellular interaction module for inputting the joint angular velocity in each direction into an angular velocity cell network to obtain a first pulse sequence representing the change in the joint angular velocity in each direction, and for inputting the target matrix into a spatial view. A second pulse sequence representing the pulse firing frequency of head-facing neurons associated with spatial view neurons is obtained in the cell network, wherein the spatial view neurons refer to computational units in the spatial view cell network, and the head-facing neurons refer to computational units in the head-facing cell network; a second type of brain multi-cell interaction module is used to input the first pulse sequence into a first intermediary cell network to obtain a first electrical signal representing the input energy of the head-facing neurons, and to input the second pulse sequence into a second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-facing neurons; a fusion calibration module is used to input the first electrical signal and the second electrical signal into the head-facing cell network to obtain the calculation result of the head posture of the target object.

[0134] In one possible implementation, the step of processing the visual data within the target time period along the time direction to obtain the target matrix includes: converting the visual data within the target time period into a first matrix and a second matrix according to preset event types, wherein the event types include brightness increase and brightness decrease, the first matrix represents the event frequency of brightness increase corresponding to each pixel coordinate within the target time period, and the second matrix represents the event frequency of brightness decrease corresponding to each pixel coordinate within the target time period; and performing grayscale conversion based on the first matrix and the second matrix to obtain the target matrix.

[0135] In one possible implementation, the apparatus further includes an image generation module for generating a target image based on the first matrix and the second matrix.

[0136] In one possible implementation, the step of calculating the joint angular velocity in each direction based on visual data and inertial data within the target time period includes: calibrating the inertial data within the target time period to obtain the calibration angular velocity in each direction, and calculating the standard deviation of the angular velocity in the corresponding direction based on the calibration angular velocity in each direction; determining the event angular velocity and pixel matching ratio in each direction based on the visual data within the target time period; and determining the joint angular velocity in the corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity in each direction.

[0137] In one possible implementation, determining the joint angular velocity for a corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity for each direction includes: for each direction, calculating a first ratio based on the angular velocity standard deviation and a preset maximum standard deviation; calculating a first difference based on the preset standard deviation and the first ratio; and using the product of a preset first ratio and the first difference, plus the sum of a preset second ratio, as a first weight for the corresponding direction; calculating a second ratio based on the pixel matching ratio and a preset maximum matching ratio; and using the product of the first ratio and the second ratio, plus the sum of the second ratio, as a second weight; normalizing based on the first weight and the second weight for each direction to obtain a first normalized weight and a second normalized weight for each direction; determining the first angular velocity for the corresponding direction based on the first normalized weight and the calibration angular velocity for each direction, and determining the second angular velocity for the corresponding direction based on the second normalized weight and the event angular velocity; and using the sum of the first angular velocity and the second angular velocity for each direction as the joint angular velocity for the corresponding direction.

[0138] In one possible implementation, the device further includes a verification module, configured to: determine a first motion trend for each target pixel based on visual data within the target time period, and determine a corresponding second motion trend based on the joint angular velocity; if there are a preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be valid; if there are fewer than the preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data within the target time period is determined to be invalid, and head pose estimation is performed based on valid multimodal data within historical time periods or by re-acquiring multimodal data within a new target time period.

[0139] In one possible implementation, the device further includes a pose estimation module, configured to: convert the target matrix into a target neuron state matrix according to a preset time encoding rule; calculate the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each preset spatial view template; determine the estimated head pose of the target object based on the head pose calculation result corresponding to the spatial view template with the highest similarity, wherein the estimated result is used to drive attractor state transitions in the spatial view cell network.

[0140] In one possible implementation, the device further includes a template update module, configured to: update the preset spatial view template if the comparison between the calculated head pose determined based on multimodal data within the target time period and the calculated head pose determined based on multimodal data within the previous time period meets a preset condition, and the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset first threshold; and update the preset spatial view template if the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset second threshold.

[0141] In one possible implementation, the angular velocity cell network comprises a linear continuous attractor network, the spatial view cell network comprises a loop continuous attractor network, and the head orientation cell network comprises a loop continuous attractor network.

[0142] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] This disclosure also provides a head pose estimation device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0144] This disclosure also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0145] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0146] Figure 5 A block diagram of a head pose estimation apparatus provided in an embodiment of this disclosure is shown. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0147] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0148] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0149] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0150] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0151] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0152] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0153] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0154] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0156] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A head pose estimation method, characterized in that, The method includes: Acquire multimodal data of the target object within a target time period, wherein the multimodal data refers to visual data and inertial data collected by a brain-like visual sensor installed on the target object; The visual data within the target time period is processed along the time direction to obtain the target matrix, and the joint angular velocity in each direction is calculated based on the visual data and inertial data within the target time period. The joint angular velocity in each direction is input into the angular velocity cell network to obtain a first pulse sequence representing the change in the joint angular velocity in each direction, and the target matrix is ​​input into the spatial view cell network to obtain a second pulse sequence representing the pulse firing frequency of the head-facing neurons associated with the spatial view neurons, wherein the spatial view neurons refer to the computational units in the spatial view cell network, and the head-facing neurons refer to the computational units in the head-facing cell network. The first pulse sequence is input into the first intermediary cell network to obtain a first electrical signal representing the input energy of the head-oriented neuron, and the second pulse sequence is input into the second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-oriented neuron; The first electrical signal and the second electrical signal are input into the head orientation cell network to obtain the calculation result of the head posture of the target object.

2. The method according to claim 1, characterized in that, The process of processing the visual data within the target time period along the time direction to obtain the target matrix includes: According to preset event types, the visual data within the target time period is converted into a first matrix and a second matrix. The event types include brightness increase and brightness decrease. The first matrix represents the event frequency of brightness increase corresponding to each pixel coordinate within the target time period, and the second matrix represents the event frequency of brightness decrease corresponding to each pixel coordinate within the target time period. The target matrix is ​​obtained by converting the first matrix and the second matrix into grayscale.

3. The method according to claim 2, characterized in that, The method further includes: Generate a target image based on the first matrix and the second matrix.

4. The method according to claim 1, characterized in that, The calculation of the joint angular velocity in each direction based on visual and inertial data within the target time period includes: The inertial data within the target time period is calibrated to obtain the calibrated angular velocity in each direction, and the standard deviation of the angular velocity in the corresponding direction is calculated based on the calibrated angular velocity in each direction. The event angular velocity and pixel matching ratio for each direction are determined based on the visual data within the target time period. The joint angular velocity in each direction is determined based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity for each direction.

5. The method according to claim 4, characterized in that, The determination of the joint angular velocity in the corresponding direction based on the event angular velocity, pixel matching ratio, angular velocity standard deviation, and calibration angular velocity in each direction includes: For each direction, a first ratio is calculated based on the standard deviation of the angular velocity and the preset maximum standard deviation, and a first difference is calculated based on the preset standard deviation and the first ratio. The product of the preset first ratio and the first difference, and the sum of the preset second ratio are used as the first weight for the corresponding direction. The second ratio is calculated based on the pixel matching ratio and the preset maximum matching ratio, and the product of the first ratio and the second ratio and the sum of the second ratio are used as the second weight. Normalization is performed based on the first weight and the second weight for each direction to obtain the first normalized weight and the second normalized weight for each direction; The first angular velocity in each direction is determined based on the first normalized weight and the calibrated angular velocity, and the second angular velocity in each direction is determined based on the second normalized weight and the event angular velocity. The sum of the first and second angular velocities in each direction is taken as the joint angular velocity in the corresponding direction.

6. The method according to claim 1, characterized in that, The method further includes: The first motion trend of each target pixel is determined based on the visual data within the target time period, and the corresponding second motion trend is determined based on the joint angular velocity. If there are a first motion trend that is the same as the second motion trend for a number of target pixels that is greater than or equal to a preset number, then the multimodal data within the target time period is determined to be valid. If there are fewer than the preset number of target pixels whose first motion trend is the same as the second motion trend, then the multimodal data in the target time period is determined to be invalid, and head pose estimation is performed based on the valid multimodal data in the historical time period or by re-acquiring the multimodal data in the new target time period.

7. The method according to claim 1, characterized in that, The method further includes: According to the preset time encoding rules, the target matrix is ​​converted into a target neuron state matrix; Calculate the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each preset spatial view template; The estimated head pose of the target object is determined based on the head pose calculation result corresponding to the spatial view template with the highest similarity. The estimated result is used to drive the attractor state transition in the spatial view cell network.

8. The method according to claim 7, characterized in that, The method further includes: If the comparison between the head pose calculation result determined by the multimodal data within the target time period and the head pose calculation result determined by the multimodal data within the previous time period meets the preset conditions, and the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than the preset first threshold, then the preset spatial view template is updated. If the similarity between the target neuron state matrix and the template neuron state matrix corresponding to each spatial view template is lower than a preset second threshold, then the preset spatial view template is updated.

9. The method according to any one of claims 1 to 8, characterized in that, The angular velocity cell network includes a linear continuous attractor network, the spatial view cell network includes a loop continuous attractor network, and the head orientation cell network includes a loop continuous attractor network.

10. A head pose estimation device, characterized in that, The device includes: The data acquisition module is used to acquire multimodal data of the target object within a target time period. The multimodal data refers to visual data and inertial data collected by a brain-like visual sensor set on the target object. The data preprocessing module is used to process the visual data within the target time period along the time direction to obtain the target matrix, and to calculate the joint angular velocity in each direction based on the visual data and inertial data within the target time period. The first type of brain multi-cell interaction module is used to input the joint angular velocity of each direction into the angular velocity cell network to obtain a first pulse sequence representing the change of the joint angular velocity in each direction, and to input the target matrix into the spatial view cell network to obtain a second pulse sequence representing the pulse firing frequency of the head-facing neurons associated with the spatial view neurons, wherein the spatial view neurons refer to the computing units in the spatial view cell network, and the head-facing neurons refer to the computing units in the head-facing cell network; The second type of brain multi-cell interaction module is used to input the first pulse sequence into the first intermediary cell network to obtain a first electrical signal representing the input energy of the head-oriented neuron, and to input the second pulse sequence into the second intermediary cell network to obtain a second electrical signal for calibrating the absolute orientation of the head-oriented neuron; The fusion calibration module is used to input the first electrical signal and the second electrical signal into the head orientation cell network to obtain the calculation result of the head posture of the target object.

Images