A neural-paradigm visual sensing vr scene dynamic perception method

Abstract

This invention discloses a VR scene dynamic perception method based on neuromorphic visual sensing, comprising: responding to an asynchronous event stream output by a neuromorphic visual sensor, mapping each event to a four-dimensional spatiotemporal coordinate system; performing recursive spatiotemporal clustering on all events based on the spatial coordinate dispersion and timestamp dispersion of each event to generate multiple initial spatiotemporal event clusters; calculating the drift trajectory of the centroid of each initial spatiotemporal event cluster in the four-dimensional spatiotemporal coordinate system, and dividing the drift trajectory into multiple continuous trajectory segments; using each trajectory segment as a dynamic perception unit, and identifying dynamic objects in the VR scene corresponding to each dynamic perception unit. This invention achieves efficient perception and recognition of dynamic objects in VR scenes through spatiotemporal joint clustering and centroid drift trajectory segmentation.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality and neuromorphic visual sensing technology, specifically a method for dynamic perception of VR scenes using neuromorphic visual sensing. Background Technology

[0002] VR scene dynamic perception methods require real-time capture and identification of dynamic changes in the scene to drive interactive responses in the virtual environment. Traditional visual perception schemes rely on image sequences output by fixed-frame-rate cameras, analyzing each frame to detect moving objects. This synchronous sampling mechanism still generates a large amount of redundant data when no motion occurs in the scene, consuming transmission bandwidth and computing resources. In cases of high-speed or sudden motion of dynamic objects, motion information between fixed frame rates is smoothed or lost, leading to response delays and trajectory breaks in dynamic perception. Event cameras, as neuromorphic visual sensors, asynchronously output only pixel-level events triggered by brightness changes, eliminating data redundancy at the source and achieving microsecond-level temporal resolution.

[0003] However, converting asynchronous event streams into dynamic object perception results that can be directly used by VR systems faces two challenges: clustering difficulties and trajectory tracking instability. Asynchronous events arrive in the form of discrete point clouds, lacking frame structure. How to reorganize effective event groups belonging to the same dynamic object in a sparse and noisy environment is a key problem to be solved. The event triggering mechanism is highly dependent on scene texture and relative motion. The event streams generated by the continuous motion of the same object are irregularly distributed in space and time. How to stably extract motion trajectory segments from these irregular distributions and identify specific dynamic objects accordingly is another key challenge. Summary of the Invention

[0004] This invention provides a VR scene dynamic perception method based on neuromorphic visual sensing. It aims to directly construct a spatiotemporal joint clustering mechanism on asynchronous event stream data to form an event cluster structure that matches the dynamic objects in the scene. By segmenting and analyzing the centroid drift trajectory, continuous motion is adaptively decomposed into identifiable dynamic perception units, eliminating the dependence on a fixed frame rate and achieving low-latency, high-temporal-resolution VR scene dynamic object perception.

[0005] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a VR scene dynamic perception method based on neuromorphic visual sensing. The method responds to the asynchronous event stream output by the neuromorphic visual sensor, mapping each event to a four-dimensional spatiotemporal coordinate system containing three spatial dimensions and one temporal dimension. The three spatial dimensions correspond to the three-dimensional spatial coordinates in the VR scene, and the temporal dimension corresponds to the timestamp of the event. Based on the dispersion of the spatial coordinates and timestamp of each event in the four-dimensional spatiotemporal coordinate system, recursive spatiotemporal clustering is performed on all events to generate multiple initial spatiotemporal event clusters. For each initial spatiotemporal event cluster, the drift trajectory of its centroid in the four-dimensional spatiotemporal coordinate system is calculated, and this drift trajectory is divided into multiple trajectory segments that are continuous both temporally and spatially. Each trajectory segment is treated as an independent dynamic perception unit, and the dynamic object corresponding to each dynamic perception unit in the VR scene is identified.

[0006] As a preferred embodiment of the present invention, in the process of generating initial spatiotemporal event clusters through recursive spatiotemporal clustering, the spatiotemporal composite distance between any two events is calculated. This spatiotemporal composite distance is obtained by weighted summing the Euclidean distance between the two events in the three spatial dimensions and the absolute difference between the two events in the time dimension. Using each event as an initial cluster center, all events with a spatiotemporal composite distance less than a dynamic threshold to this initial cluster center are grouped into a temporary cluster. Then, the events that have already been grouped are removed from all events, and a new initial cluster center is selected from the remaining events. This grouping operation is repeated until all events are grouped into at least one temporary cluster. Finally, overlapping temporary clusters are merged, and each merged temporary cluster is used as an initial spatiotemporal event cluster. This clustering method effectively aggregates events similar in the spatiotemporal dimensions, improving the accuracy of subsequent dynamic object segmentation.

[0007] Preferably, the weighting coefficients used in the above-mentioned spatiotemporal composite distance can be adaptively adjusted according to the movement speed of dynamic objects in the VR scene, so that the clustering results can maintain sensitivity to spatiotemporal related events when the movement speed of objects in the scene changes.

[0008] As another preferred technical solution of the present invention, in the process of calculating the centroid drift trajectory of the initial spatiotemporal event cluster and dividing it into multiple continuous trajectory segments, all events within an initial spatiotemporal event cluster are traversed in ascending order of timestamps. The average spatial coordinates of all events within each traversed time cluster in the three spatial dimensions are calculated sequentially, and this average value is used as the instantaneous centroid of the initial spatiotemporal event cluster at that moment. The instantaneous centroids of all moments are connected in chronological order to form the initial centroid drift trajectory of the initial spatiotemporal event cluster. The spatial displacement between two adjacent instantaneous centroids on the initial centroid drift trajectory is calculated. The connection point between adjacent instantaneous centroids whose spatial displacement exceeds a preset spatial threshold is determined as a dividing point, and the initial centroid drift trajectory is divided into multiple continuous trajectory segments based on the dividing point. This segmentation method based on the abrupt change in centroid spatial displacement can accurately separate event clusters belonging to different motion stages.

[0009] Preferably, the aforementioned preset spatial threshold is dynamically calculated based on the time interval between adjacent instantaneous centroids on the initial centroid drift trajectory, in order to adapt to the motion segmentation requirements at different time scales.

[0010] As a further improvement of this invention, when identifying dynamic objects corresponding to a dynamic sensing unit, for a trajectory segment, its corresponding time interval and spatial variation range are obtained. From the asynchronous event stream output by the neuromorphic visual sensor, events whose timestamps fall within the time interval and whose spatial coordinates fall within the spatial variation range are selected to form a set of associated events corresponding to the trajectory segment. The pulse frequency modulation features and pulse density modulation features of all events in this set of associated events are calculated, and these two types of features are input into a pre-trained object classifier, which outputs the object category of the dynamic object corresponding to the trajectory segment. By combining the frequency and density modulation features of the events, the accuracy and robustness of identifying dynamic objects represented by neuromorphic data are improved.

[0011] Preferably, the object classifier is a classifier built on a spiking neural network, whose number of input layer neurons matches the total dimension of the pulse frequency modulation feature and the pulse density modulation feature, thus being able to well adapt to the pulse signal form naturally output by the neuromorphic visual sensor.

[0012] Specifically, when calculating the pulse frequency modulation (PFM) and pulse density modulation (PDM) features of a set of related events, the polarity flag is extracted for each event in the set. Events with the same polarity flag are grouped into a subset based on their timestamps from smallest to largest. The reciprocal of the time interval between adjacent events within each subset is calculated, and the average of all calculated reciprocals is used as the PFM feature of that subset. Simultaneously, the ratio of the number of events within a preset duration window to the width of that window is used as the PFM feature of the set. This method characterizes the texture of the event flow from both the polarity and spatiotemporal density dimensions, and the feature extraction method closely matches the output characteristics of the event camera.

[0013] In a further preferred embodiment, the width of the aforementioned preset duration window is dynamically and adaptively adjusted according to the frame rate of the VR scene, so that the calculation of the pulse density modulation feature can smoothly change with the scene refresh rate.

[0014] In a preferred embodiment of the present invention, after generating multiple initial spatiotemporal event clusters and before calculating the drift trajectory, a static background filtering operation is performed. Specifically, a preset static background space region in the VR scene is obtained, and it is determined whether an initial spatiotemporal event cluster contains more than a preset proportion of events falling within this static background space region. When an initial spatiotemporal event cluster contains more than a preset proportion of events falling within this static background space region, the initial spatiotemporal event cluster is marked as a static background event cluster, and this static background event cluster is skipped in the subsequent centroid drift trajectory calculation step. This method can effectively filter out the interference of static background events and significantly reduce the computational overhead of subsequent processing.

[0015] In another preferred embodiment of the present invention, after generating multiple initial spatiotemporal event clusters and before calculating the drift trajectory, a cluster merging operation based on spatiotemporal overlap is performed. Specifically, the initial spatial distance between the centroids of any two initial spatiotemporal event clusters and the temporal overlap length between the timestamp coverage areas of these two initial spatiotemporal event clusters are calculated. When the initial spatial distance between two initial spatiotemporal event clusters is less than a first merging threshold and their temporal overlap length is greater than a second merging threshold, the two initial spatiotemporal event clusters are merged into a composite event cluster, and this composite event cluster replaces the original two initial spatiotemporal event clusters in the subsequent calculation of the centroid drift trajectory. Through this merging, event clusters of the same object that have been fragmented by short-term occlusion or interference can be re-aggregated.

[0016] As a further preferred embodiment of the present invention, after identifying the dynamic object corresponding to the dynamic sensing unit, the interactive attention of the dynamic object is also determined. Specifically, based on the trajectory segment of a dynamic sensing unit, its motion trend vector in the VR scene is obtained. This motion trend vector is obtained by pointing from the instantaneous centroid of the endpoint of the trajectory segment to the instantaneous centroid of the starting point of the trajectory segment. The current orientation vector of a preset virtual camera in the VR scene is obtained, and the spatial angle between the motion trend vector and the current orientation vector is calculated. When the spatial angle is less than a preset attention threshold, the dynamic object corresponding to the dynamic sensing unit is marked as the interactive object that needs to be focused on. Thus, the system can automatically highlight or prompt objects whose motion orientation is related to its own perspective to the user, improving the naturalness of the interaction.

[0017] Preferably, when acquiring the motion trend vector, a preset time window is extracted at the end position of a trajectory segment, and the average position of the centroid of the dynamic sensing unit within this time window is calculated as the end reference position; a preset time window is extracted at the start position of the trajectory segment, and the average position of the centroid of the dynamic sensing unit within this time window is calculated as the start reference position; the vector pointing from the end reference position to the start reference position is used as the preliminary motion trend vector, and this vector is normalized to obtain the motion trend vector of the dynamic sensing unit. This window smoothing method overcomes the noise caused by single event points, making the motion trend judgment more stable and reliable.

[0018] As another preferred embodiment of the present invention, after identifying dynamic objects, semantic binding and parameter correction are performed based on an object model library. Specifically, the identification results of all dynamic perception units in the VR scene are obtained, and each dynamic object in the identification results is matched with a model in the virtual object model library established in the VR scene. When a dynamic object matches a unique virtual object model, the dynamic attribute parameters of the virtual object model are bound to the trajectory segment of the dynamic perception unit. These dynamic attribute parameters include the size constraint range and joint motion range of the virtual object model. Subsequently, the joint motion range of the virtual object model is corrected according to the actual shape of the trajectory segment of the dynamic perception unit. This method feeds back the real motion information perceived by neuromorphic vision to the virtual model, making the motion performance of the virtual object more consistent with the real scene.

[0019] The technical effects and advantages provided by the present invention in the above technical solution are as follows: When performing recursive spatiotemporal clustering on all asynchronous events, a spatiotemporal composite distance metric that combines Euclidean distance in the spatial dimension and absolute difference in the time dimension is adopted. The weighting coefficients in this distance metric are adaptively adjusted according to the movement speed of dynamic objects in the VR scene.

[0020] When a dynamic object moves at high speed, the weight of the time dimension adaptively decreases, while the dominance of the spatial dimension increases, allowing events generated by the same fast-moving object over a longer time span to still be accurately clustered. When a dynamic object moves at low speed or is stationary, the weight of the time dimension adaptively increases, allowing spatially adjacent events within the same neighborhood at similar times to be finely distinguished. Events already assigned to temporary clusters are recursively removed, and new cluster centers are selected. Combined with dynamic threshold discrimination, this process peels away discrete but spatiotemporally related event groups from the sparse event stream layer by layer, forming independent clusters.

[0021] Overlapping temporary clusters are eliminated through merging operations, resulting in an initial spatiotemporal event cluster that corresponds to the event triggering patterns of the actual dynamic object in terms of both spatial compactness and temporal continuity. This provides a complete set of events with clear boundaries for subsequent centroid drift analysis. Events within the initial spatiotemporal event cluster are traversed sequentially, and the average spatial coordinates are calculated at each time step to form the initial centroid drift trajectory. The spatial displacement between adjacent instantaneous centroids is then calculated and compared with a preset spatial threshold dynamically calculated based on the time interval between adjacent instantaneous centroids to determine the segmentation point.

[0022] Where time intervals are longer, the dynamically generated spatial threshold is correspondingly relaxed to allow centroid jumps caused by occlusion or event quiet periods to be prevented from being missegmented; where time intervals are dense, the spatial threshold is tightened to remain sensitive to subtle trajectory deflections. This adaptive threshold segmentation method allows the centroid drift trajectory in a continuous motion containing acceleration, deceleration, and turning to be naturally divided into multiple near-uniform straight lines or smoothly curved trajectory segments, each corresponding to a relatively consistent stage of motion.

[0023] Using trajectory segments as dynamic sensing units, events within time intervals and spatial variation ranges are selected to form a set of associated events. Pulse frequency modulation features and pulse density modulation features are extracted, and an object classifier based on a spiking neural network outputs the object category. The set of associated events faithfully reflects the complete spatiotemporal distribution of events in this motion phase. The pulse frequency modulation features finely characterize the temporal structure of the sensor on positive and negative event triggering by using the inverse mean of the time intervals between adjacent events in the polarity event subset. The pulse density modulation features reflect the density of event triggering by using the ratio of the number of events in the duration window to the window width. The combination of these two features enables the classifier to respond simultaneously to changes in triggering patterns caused by texture differences and density changes caused by motion speed, achieving reliable determination of dynamic object categories without the need to recover grayscale images or depth maps. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart of a VR scene dynamic perception method using neuromorphic visual sensing; Figure 2 It is a flowchart of recursive spatiotemporal event clustering based on adaptive weighting coefficients; Figure 3 This is a flowchart of the calculation and segmentation process for the centroid drift trajectory of the initial spatiotemporal event cluster; Figure 4 This is a flowchart of the static background event cluster tagging and event cluster merging process; Figure 5 This is a schematic diagram of the distribution of the initial spatiotemporal event cluster on a two-dimensional spatial plane; Figure 6 It is a diagram of the centroid drift trajectory of the initial spacetime event cluster. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See Figure 1 This invention provides a VR scene dynamic perception method based on neuromorphic visual sensing. The method includes: responding to an asynchronous event stream output by a neuromorphic visual sensor, mapping each event to a four-dimensional spatiotemporal coordinate system, wherein the three spatial dimensions of the four-dimensional spatiotemporal coordinate system correspond to the three-dimensional spatial coordinates in the VR scene, and the one time dimension corresponds to the timestamp of the event occurrence; performing recursive spatiotemporal clustering on all events based on the discreteness of the spatial coordinates and the discreteness of the timestamp of each event in the four-dimensional spatiotemporal coordinate system to generate multiple initial spatiotemporal event clusters; calculating the drift trajectory of the centroid of each initial spatiotemporal event cluster in the four-dimensional spatiotemporal coordinate system, and dividing the drift trajectory into multiple continuous trajectory segments; using each trajectory segment as a dynamic perception unit, and identifying the dynamic objects in the VR scene corresponding to each dynamic perception unit.

[0028] Example 1: In specific implementation, please refer to Figure 2The process of generating multiple initial spatiotemporal event clusters by performing recursive spatiotemporal clustering on all events requires calculating the spatiotemporal composite distance between any two events. Let the event stream contain N events, each represented as... ,in, , , The three-dimensional spatial coordinates of the event in the VR scene. The timestamp of the event. This is the polarity flag for the event. For any two events... and Euclidean distance in three spatial dimensions for: Among them, the absolute difference in the time dimension for: Among them, spatiotemporal composite distance The formula is as follows: The Euclidean distance in space is obtained by weighted summation of the absolute difference in time. in, These are spatial weighting coefficients. The time-weighted coefficient, and satisfy . For the event With the event The spatiotemporal composite distance between them Let Euclidean distance be the distance between two events in three spatial dimensions. This represents the absolute difference between two events over time. Spatial weighting coefficient. and time weighting factor The motion speed is adaptively adjusted based on the movement speed of dynamic objects in the VR scene. The motion speed is estimated by analyzing the instantaneous centroid displacement of event clusters within a window over a recent period. Before each recursive spatiotemporal clustering operation, a preliminary spatial division is performed on the latest predetermined number of events arriving in the asynchronous event stream. The average timestamp change rate of events in each sub-region is calculated, and the maximum average timestamp change rate across all sub-regions is taken as the reference motion speed of the dynamic objects in the current scene. Then determine according to the following rules. and :when Less than or equal to the preset low speed threshold When, set , ;when Greater than or equal to the preset high-speed threshold When, set , ;when Between and When the interval is between, linear interpolation is used for calculation. : and order Among them, the preset low-speed threshold and preset high-speed threshold The speed range of typical interactive objects in a VR system is preset in advance, for example, by setting the speed range of movement in advance. Set to 0.2 m / s, The speed was set to 2.0 m / s. This setting is based on the following: when the object's speed is below 0.2 m / s, the object can be considered to be in a quasi-stationary state, with extremely slow changes in spatial position, and spatial clustering should dominate; when the object's speed is above 2.0 m / s, the object is in a state of rapid movement, and continuity in the time dimension is more important for clustering.

[0029] After determining the weighting coefficients in the spatiotemporal composite distance, recursive clustering is performed using each event as an initial cluster center. An event that has not yet been assigned to any temporary cluster is selected as the current initial cluster center. .calculate The spatiotemporal composite distance between the event and all other events that have not yet been assigned to any temporary cluster. A dynamic threshold is set. To satisfy all The incident and They were all grouped into a temporary cluster. Medium. Dynamic threshold The method for determining this is as follows: calculate the median of the spatiotemporal composite distances between all pairs of events that have not yet been assigned to the temporary cluster, and multiply this median by a fixed scaling factor. As a dynamic threshold, where The value range is from 0.3 to 0.6. Using the median value can effectively avoid the influence of a few extreme distance values ​​on the threshold. The value of 0.3 to 0.6 is based on the fact that this range can cluster spatially and temporally closely related events into one cluster while avoiding forcibly including loosely related events into the same cluster. Events already assigned to a temporary cluster will be grouped together. All events are removed from the set of pending events. Among the remaining pending events, an event that has not yet been assigned to any temporary cluster is selected as the new initial cluster center. The above operations of calculating the spatiotemporal composite distance, comparing it with the dynamic threshold, and assigning it to a temporary cluster are repeated until all events are assigned to at least one temporary cluster.

[0030] Since an event may be assigned to multiple temporary clusters generated with different initial cluster centers, meaning there is overlap between temporary clusters, it is necessary to merge these overlapping temporary clusters. Specifically, an undirected graph is constructed, where each node represents a temporary cluster. If two temporary clusters contain at least one identical event, an edge is added between the corresponding two nodes. All connected components in the undirected graph are identified, and all temporary clusters contained in each connected component are merged into an initial spatiotemporal event cluster. During merging, all events belonging to the same initial spatiotemporal event cluster are deduplicated to ensure that each event is recorded only once in the initial spatiotemporal event cluster. Through this process, multiple initial spatiotemporal event clusters can be generated, each corresponding to a set of events that are closely related in both spatial and temporal dimensions.

[0031] See Figure 5 The figure illustrates the distribution of the five initial spatiotemporal event clusters generated by the recursive spatiotemporal clustering method in Example 1 along spatial dimensions X and Y in a two-dimensional plane. The horizontal axis represents spatial dimension X in meters, ranging from approximately 1.5 meters to 10 meters; the vertical axis represents spatial dimension Y in meters, ranging from approximately 1.5 meters to 9 meters. The legend identifies the five different initial spatiotemporal event clusters using circles, squares, triangles, rhombuses, and inverted triangles, respectively.

[0032] As shown in the figure, the five initial spatiotemporal event clusters exhibit a relatively compact and mutually separated aggregation state in the spatial dimension. Event points within each cluster are concentrated, while a certain spatial distance is maintained between clusters. This indicates that recursive spatiotemporal clustering effectively distinguishes spatially non-overlapping event sets through spatiotemporal composite distance. Specifically, initial spatiotemporal event cluster 1 is clustered within the range of approximately 1.5 to 2.5 meters on the X-axis and approximately 2.5 to 3.5 meters on the Y-axis; initial spatiotemporal event cluster 2 is concentrated in the region of approximately 6 to 7.5 meters on the X-axis and approximately 1.5 to 2.5 meters on the Y-axis; initial spatiotemporal event cluster 3 is located within the range of approximately 3.5 to 4.5 meters on the X-axis and approximately 7.5 to 8.5 meters on the Y-axis; initial spatiotemporal event cluster 4 is distributed in the region of approximately 8.5 to 9.5 meters on the X-axis and approximately 6.5 to 7.5 meters on the Y-axis; and initial spatiotemporal event cluster 5 is located in the interval of approximately 4.5 to 6.0 meters on the X-axis and approximately 4.0 to 6.0 meters on the Y-axis.

[0033] The spatial distribution of each event cluster in the figure conforms to the clustering result described in Example 1, which is based on the weighted calculation of spatiotemporal composite distance using spatial Euclidean distance and temporal absolute difference. The aggregation of event points in the spatial dimension reflects the use of spatial weighting coefficients. Effective use of spatial distance and dynamic thresholds Reasonable setting of cluster boundaries. The obvious spatial intervals between event clusters avoid excessive overlap of event clusters, which is consistent with the strategy of merging overlapping temporary clusters of undirected graph connected components in the embodiment, and ensures the accurate division of the final initial spatiotemporal event clusters.

[0034] In summary, the figure accurately reflects the spatial distribution characteristics of multiple initial spatiotemporal event clusters generated by the recursive spatiotemporal clustering step in Example 1, and verifies the event cluster partitioning effect achieved by spatiotemporal composite distance weighting and dynamic threshold control. This provides a solid spatial foundation for dynamic object recognition based on the centroid drift trajectory of event clusters in subsequent examples.

[0035] Example 2: In specific implementation, please refer to Figure 3 For an initial spatiotemporal event cluster, the process of calculating the drift trajectory of the centroid of the initial spatiotemporal event cluster in a four-dimensional spatiotemporal coordinate system is as follows: Obtain all events within the initial spatiotemporal event cluster and arrange them in ascending order of timestamps. Iterate through every moment within the time range covered by the initial spatiotemporal event cluster. This time range is determined by the earliest and latest timestamps in the initial spatiotemporal event cluster. The step size of the traversal is set to an integer multiple of the minimum time interval between two adjacent events to ensure that at least one event occurs at each moment. At each traversed moment... Filter out timestamps equal to from the initial spatiotemporal event cluster. For all events, calculate the mean of the coordinates of these events in the three spatial dimensions to obtain the instantaneous centroid. Instantaneous center of mass spatial coordinates Determined by the following formula: in, The timestamp in the initial spacetime event cluster is equal to A set of events, For set The number of events in the middle, , , For the event The three-dimensional spatial coordinates are determined. The instantaneous centroids corresponding to all moments covered by the initial spatiotemporal event cluster are connected in chronological order to form the initial centroid drift trajectory of the initial spatiotemporal event cluster. The initial centroid drift trajectory consists of a series of instantaneous centroid nodes arranged in ascending chronological order, with adjacent nodes connected by straight line segments. After obtaining the initial centroid drift trajectory, the trajectory segments are divided. The spatial displacement between two adjacent instantaneous centroids on the initial centroid drift trajectory is calculated. and Spatial displacement Calculate as follows: in, , , instantaneous center of mass spatial coordinates, , , instantaneous center of mass Spatial coordinates. A preset spatial threshold is set, and each spatial displacement... Compare with a preset space threshold. When When the instantaneous centroid exceeds the preset spatial threshold, With instantaneous center of mass The connection points between points are used as dividing points. Based on all dividing points, the initial centroid drift trajectory is divided into multiple continuous trajectory segments. Each trajectory segment is formed by sequentially connecting all instantaneous centroids between two adjacent dividing points, or by the instantaneous centroids between the trajectory start point and the first dividing point, and between the last dividing point and the trajectory end point, respectively. The preset spatial threshold is determined using the following dynamic calculation method, and the calculation formula is: in, The preset spatial threshold obtained from the current calculation is applicable to adjacent instantaneous centroids. and between. The base threshold is a pre-defined spatial distance benchmark value, determined based on the spatial resolution of the neuromorphic vision sensor in the VR scene. The spatial resolution is denoted as... Basic threshold Set as The basis for setting the spatial resolution to 3 times is that when the displacement of the centroid exceeds the spatial scale corresponding to three pixels in adjacent moments, it can be considered that the object has undergone a significant positional jump in space. This jump often means a sudden change in the direction of motion or a switch of the object. The scaling factor is used to linearly map time intervals to spatial distance increments. The value is determined by analyzing the maximum possible motion speed of dynamic objects in a VR scene. Confirmed, Order Maximum possible speed of motion Determined based on the limits of human movement set by the VR application, for example, Set to 5 meters per second. For two adjacent instantaneous centroids on the initial centroid drift trajectory and The time interval between them is obtained by subtracting the timestamps. Through the above process, for each initial spatiotemporal event cluster, multiple continuous trajectory segments can be obtained by dividing its centroid drift trajectory.

[0036] See Figure 6 The vertical axis of the figure represents the spatial coordinate X (unit: meters), and the horizontal axis represents time (unit: milliseconds), showing the drift trajectories of the centroids of five initial spatiotemporal event clusters along the X-axis in a four-dimensional spatiotemporal coordinate system. The curves are distinguished by different line types: solid lines correspond to the centroid trajectory of initial spatiotemporal event cluster 1, dashed lines to event cluster 2, dotted lines to event cluster 3, dotted lines to event cluster 4, and sparse dotted lines to event cluster 5. The figure also marks the locations of dividing points, indicating key time nodes for segmenting the centroid drift trajectories according to the preset spatial threshold set in Example 2.

[0037] As shown in the figure, the five trajectory curves exhibit relatively smooth and continuous changes within the time interval of 0 to 1000 milliseconds, reflecting the dynamic drift process of the centroid of the initial spatiotemporal event cluster in the X-axis direction. Specifically, the overall X-coordinate variation range of the centroid trajectory of event cluster 1 is approximately between 0.5 meters and 1.6 meters. The curve initially shows a slow upward trend followed by a gradual decline, reflecting the spatial position change pattern of the dynamic object corresponding to this event cluster within this time period. The centroid trajectories of event clusters 2 to 5 are all located in a higher X-coordinate range, and the trajectory curves show different fluctuation trends, indicating that the dynamic objects in each event cluster have obvious hierarchical distribution and dynamic rotation or movement characteristics in space.

[0038] The segmentation points on each trajectory are set according to the dynamic threshold calculation rules in Example 2, clearly dividing multiple continuous trajectory segments of the centroid drift trajectory. The segmentation points are mostly concentrated at locations where the trajectory exhibits significant spatial jumps, reflecting significant changes in the direction or state of motion of the dynamic object. Through this trajectory segmentation, the process of dividing a continuous initial spatiotemporal event cluster into multiple dynamic sensing units is effectively realized, laying the foundation for subsequent dynamic object recognition.

[0039] In summary, the centroid X-coordinate drift trajectories and their segmentation points of the five initial spatiotemporal event clusters illustrated in the figure intuitively reflect the technical solution and effect of calculating the centroid drift trajectory and segmenting the trajectory in Example 2. It demonstrates the dynamic change characteristics of different dynamic objects in the time and space dimensions and the rationality and effectiveness of the trajectory segmentation strategy.

[0040] Example 3: See Figure 4After generating multiple initial spatiotemporal event clusters and before calculating the centroid drift trajectory of each initial spatiotemporal event cluster, the marking and skipping of static background event clusters is performed. A pre-defined static background space region in the VR scene is obtained. This region is pre-defined by the VR scene developer during scene creation and is a set of fixed regions in 3D space used to describe background elements such as walls, floors, and fixed decorations that do not change position during scene execution. For each initial spatiotemporal event cluster, the 3D spatial coordinates of all events within the cluster are counted, and the number of events whose spatial coordinates fall within the static background space region is recorded as follows: The total number of events within the initial spatiotemporal event cluster is denoted as... Calculate the proportion value. as follows: in, This represents the number of events in the initial spatiotemporal event cluster whose spatial coordinates fall within the static background spatial region. This represents the total number of events in the initial spatiotemporal event cluster. The calculated proportion... Compared with the preset ratio threshold Compare the results. Preset ratio threshold. The threshold is set to 0.8. The reason for setting it to 0.8 is that when more than four-fifths of the events in the initial spatiotemporal event cluster originate from the static background space region, it can be determined that the initial spatiotemporal event cluster as a whole corresponds to the static background object rather than the dynamic object. This threshold can effectively identify the static background cluster while retaining the boundary events of the dynamic object.

[0041] When an initial spatiotemporal event cluster Greater than the preset ratio threshold When this initial spatiotemporal event cluster is marked as a static background event cluster, it is skipped directly in the subsequent centroid drift trajectory calculation step. Drift trajectories and trajectory segments are not generated for static background event clusters to reduce computational resource consumption and avoid misidentifying static backgrounds as dynamic objects.

[0042] Simultaneously with the above processing, a merging process between initial spatiotemporal event clusters is also performed. The initial spatial distance between the centroids of any two initial spatiotemporal event clusters not marked as static background event clusters, and the temporal overlap length between the timestamp coverage areas of any two initial spatiotemporal event clusters are calculated. For two initial spatiotemporal event clusters... and Calculate the initial spatiotemporal event cluster center of mass center of mass spatial coordinates For the initial spacetime event cluster The arithmetic mean of the spatial coordinates of all events within the event set in each dimension. Similarly, the initial spatiotemporal event cluster is calculated. center of mass spatial coordinates Initial spatial distance Calculated using Euclidean distance: in, For the initial spacetime event cluster The average value of all events in the X-axis space coordinates. For the initial spacetime event cluster The average value of all events in the X-axis space coordinates. , , , The corresponding meanings follow the same logic. Calculate the initial spatiotemporal event cluster. and the initial spacetime event cluster The length of time overlap between timestamp coverage areas. Initial spatiotemporal event cluster. The timestamp coverage is determined by the earliest timestamp of all its events. and latest timestamp Constituting the interval Initial spacetime event cluster The timestamp coverage is determined by the earliest timestamp of all its events. and latest timestamp Constituting the interval Time overlap length For interval and interval The intersection length. If the two intervals have no intersection, then the time overlap length. Zero. Set the first merging threshold. Second merging threshold First merging threshold Based on the spatial resolution of neuromorphic visual sensors in VR scenes Confirmed, the value is spatial resolution. Five times. The basis for taking a value of five times is that when the centroid distance between two event clusters is less than 5 spatial resolution units, in VR scene perception, these two clusters are very likely to belong to the adjacent parts of the same moving object, rather than two independent dynamic objects.

[0043] Second merging threshold Based on the typical interactive event flow time density in VR scenes, a value of 30 milliseconds is set. The rationale for choosing 30 milliseconds is that the timestamp discrepancy between consecutive events occurring on the same dynamic object typically does not exceed 30 milliseconds; two event clusters exceeding this time interval are no longer directly correlated in the time dimension. The initial spatial distance... With the first merging threshold Compare the time overlap lengths With the second merging threshold Comparison. When the initial spatial distance... Less than the first merging threshold And the length of time overlap Greater than the second merging threshold At that time, the initial spatiotemporal event cluster and the initial spacetime event cluster These are merged into a composite event cluster. The composite event cluster contains the initial spacetime event cluster. and the initial spacetime event cluster All non-repeating events are included. The original two initial spatiotemporal event clusters are replaced by composite event clusters in the subsequent calculation of the centroid drift trajectory. This merging process can reintegrate event clusters that were incorrectly segmented due to sensor noise or transient occlusion, improving the continuity of dynamic object trajectories.

[0044] For any two initial spatiotemporal event clusters that are not marked as static background event clusters, the above merging judgment and operation are performed until no new merging occurs. After the merging process is completed, the centroid drift trajectory is calculated for all retained initial spatiotemporal event clusters and composite event clusters.

[0045] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for dynamic perception of VR scenes using neuromorphic visual sensing, characterized in that, include: In response to the asynchronous event stream output by the neuromorphic visual sensor, each event is mapped to a four-dimensional spatiotemporal coordinate system. The three spatial dimensions of the four-dimensional spatiotemporal coordinate system correspond to the three-dimensional spatial coordinates in the VR scene, and the one time dimension corresponds to the timestamp of the event. Based on the spatial coordinate dispersion and timestamp dispersion of each event in the four-dimensional spatiotemporal coordinate system, recursive spatiotemporal clustering is performed on all events to generate multiple initial spatiotemporal event clusters. Calculate the drift trajectory of the centroid of each initial spatiotemporal event cluster in a four-dimensional spatiotemporal coordinate system, and divide the drift trajectory into multiple continuous trajectory segments; Each trajectory segment is used as a dynamic perception unit to identify dynamic objects in the VR scene corresponding to each dynamic perception unit.

2. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 1, characterized in that, The step of performing recursive spatiotemporal clustering on all events to generate multiple initial spatiotemporal event clusters, based on the spatial coordinate dispersion and timestamp dispersion of each event in a four-dimensional spatiotemporal coordinate system, includes: Calculate the spatiotemporal composite distance between any two events in a four-dimensional spatiotemporal coordinate system. The spatiotemporal composite distance is obtained by weighted summation of the Euclidean distance between the two events in the three spatial dimensions and the absolute difference between the two events in the time dimension. Using each event as an initial cluster center, all events whose spatiotemporal composite distance from the initial cluster center is less than a dynamic threshold are grouped into a temporary cluster; Remove events that have been assigned to temporary clusters from all the events, and reselect new initial cluster centers from the remaining events. Repeat the assignment operation until all events are assigned to at least one temporary cluster. The overlapping temporary clusters are merged, and each of the merged temporary clusters is used as an initial spatiotemporal event cluster.

3. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 2, characterized in that, The weighting coefficients in the spatiotemporal composite distance are adaptively adjusted according to the movement speed of dynamic objects in the VR scene.

4. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 2, characterized in that, The steps of calculating the drift trajectory of the centroid of each initial spatiotemporal event cluster in a four-dimensional spatiotemporal coordinate system and dividing the drift trajectory into multiple continuous trajectory segments include: According to the timestamp in ascending order, all events in an initial spatiotemporal event cluster are traversed, and the average spatial coordinates of all events in the time cluster traversed at each time are calculated in turn. The average spatial coordinates calculated at each time moment are used as the instantaneous centroid of the initial spatiotemporal event cluster at that time moment. Connect the instantaneous centroids at all moments in chronological order to form the initial centroid drift trajectory of the initial spatiotemporal event cluster; Calculate the spatial displacement between two adjacent instantaneous centroids on the initial centroid drift trajectory. Use the connection point between adjacent instantaneous centroids whose spatial displacement exceeds a preset spatial threshold as the dividing point. Divide the initial centroid drift trajectory into multiple continuous trajectory segments based on the dividing point.

5. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 4, characterized in that, The preset spatial threshold is dynamically calculated based on the time interval between adjacent instantaneous centroids on the initial centroid drift trajectory.

6. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 4, characterized in that, The steps of using each trajectory segment as a dynamic sensing unit and identifying dynamic objects in the VR scene corresponding to each dynamic sensing unit include: For a trajectory segment, obtain the corresponding time interval and spatial variation range of the trajectory segment; From the asynchronous event stream output by the neuromorphic visual sensor, events whose timestamps fall within the time interval and whose spatial coordinates fall within the range of spatial changes are selected to form a set of associated events corresponding to the trajectory segment; Calculate the pulse frequency modulation features and pulse density modulation features of all events in the associated event set, and input the pulse frequency modulation features and pulse density modulation features into a pre-trained object classifier. The object classifier then outputs the object category of the dynamic object corresponding to the trajectory segment.

7. The VR scene dynamic perception method based on neuromorphic visual sensing according to claim 6, characterized in that, The object classifier is a classifier built based on a spiking neural network, and the number of neurons in the input layer of the object classifier matches the total dimension of the pulse frequency modulation feature and the pulse density modulation feature.

8. A VR scene dynamic perception method based on neuromorphic visual sensing according to claim 6, characterized in that, The steps for calculating the pulse frequency modulation characteristics and pulse density modulation characteristics of all events in the associated event set include: For each event in the set of associated events, extract the polarity flag of the event; Based on the ascending order of timestamps, events with the same polarity flag in the associated event set are divided into a subset of polarity events; Calculate the reciprocal of the time interval between adjacent events in each polarity event subset, and use the mean of all calculated reciprocals as the pulse frequency modulation feature of the polarity event subset; The ratio of the number of events in the associated event set within a preset duration window to the width of the preset duration window is used as the pulse density modulation feature of the associated event set.

9. A VR scene dynamic perception method based on neuromorphic visual sensing according to claim 8, characterized in that, The width of the preset duration window is dynamically and adaptively adjusted according to the frame rate of the VR scene.

10. A VR scene dynamic perception method based on neuromorphic visual sensing according to claim 1, characterized in that, After the step of generating multiple initial spatiotemporal event clusters, and before the step of calculating the drift trajectory of the centroid of each initial spatiotemporal event cluster, the method further includes: Obtain a preset static background space region in the VR scene, and determine whether there are events in an initial spatiotemporal event cluster that fall into the static background space region exceeding a preset proportion; When more than a preset proportion of events in an initial spatiotemporal event cluster fall into the static background space region, the initial spatiotemporal event cluster is marked as a static background event cluster, and the static background event cluster is skipped in the subsequent centroid drift trajectory calculation step.

Images