Event-driven time-domain modeling and multi-scale based video viewpoint prediction method

Abstract

The application discloses a video viewpoint prediction method based on event-driven time domain modeling and multi-scale, which is aimed at the non-rigid deformation and topological structure change of Gaussian points in 4D GS video, utilizes multi-dimensional features to construct a composite similarity function, establishes a robust inter-frame correlation through a corresponding relationship perception alignment module, calculates the time domain change event score of the Gaussian points, uses a Gaussian mixture model (GMM) to fit the score distribution and solve the optimal decision threshold, adaptively screens dynamic salient points based on a dynamic perception threshold, and removes static redundant points, inputs the screened dynamic point sequence into a multi-scale structured state space model (SSM), captures short-time local motion and long-time time domain dependence in layers, combines the user head motion trajectory and space-time features, and predicts the future viewpoint direction, so that the efficient and accurate prediction of the user viewpoint direction in an immersive media scene is realized under the premise of ensuring real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of mobile multimedia computing and network video transmission technology, specifically to a video viewpoint prediction method based on event-driven temporal modeling and multi-scale. Background Technology

[0002] In immersive media applications, viewpoint prediction is a key technology for achieving adaptive streaming media delivery and foveated rendering. By accurately predicting the user's future viewing direction, the system can prioritize the delivery and rendering of high-quality content within the user's field of vision, thereby significantly improving the user experience with limited bandwidth.

[0003] However, most existing viewpoint prediction methods are designed for 2D panoramic video or traditional 3D point cloud video and cannot be directly applied to the emerging 4D Gaussian splatter (4DGS) video. 4DGS video has unique data characteristics: 1. Time-varying topology: Unlike traditional grids, the number and connectivity of Gaussian points change drastically over time, with points disappearing and being created, making it impossible to establish a one-to-one index relationship between frames. 2. High sparsity and irregularity: Gaussian points are extremely unevenly distributed in space, and a large number of points remain stationary for most of the time, with only a small number of areas (such as moving figures) changing.

[0004] Traditional temporal modeling methods based on RNNs or Transformers typically assume a fixed input sequence length and clear correspondences, which fails in 4DGS scenarios. Furthermore, indiscriminately updating all points frame-by-frame results in a significant waste of computational resources, making it difficult to meet the real-time requirements of VR devices. Therefore, developing an efficient temporal prediction method that can adapt to dynamic topological changes in point clouds and intelligently perceive differences between static and dynamic states is particularly important. Summary of the Invention

[0005] The video viewpoint prediction method based on event-driven temporal modeling and multi-scale proposed in this invention can at least solve one of the technical problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The video viewpoint prediction method based on event-driven temporal modeling and multi-scale includes the following steps:

[0008] S100: Acquire 4DGS video viewpoint frame data to construct the current frame point. Aligned virtual next frame features;

[0009] S200, based on the current frame point Aligned virtual next-frame features are used to calculate temporal event scores for Gaussian points and establish a distribution model;

[0010] S300. The distribution model is iteratively estimated using the expectation-maximization algorithm to solve for the optimal dynamic sensing threshold and perform adaptive screening.

[0011] S400, based on adaptively selected dynamic points, constructs a multi-scale time-domain state-space model SSM;

[0012] S500, based on the multi-scale temporal state-space model SSM, completes viewpoint prediction output;

[0013] The temporal event score calculation method in step S200 includes: for each Gaussian point at the current time, firstly, obtaining the position parameters, color parameters, and semantic features of the point in the current frame and the next frame after alignment; then, calculating the position change, color change, and semantic change of the Gaussian point in adjacent time moments respectively; and then, weighting and summing the position change, color change, and semantic change according to preset weights to obtain the temporal event score of the Gaussian point.

[0014] Furthermore, in step S100 of the present invention, the current frame point is constructed. Aligned virtual next-frame feature methods include:

[0015] Extract the spatial features of Gaussian points in adjacent frames, where the spatial features include the positions and geometric coordinates of Gaussian points in the 4DGS video viewpoint frame data. The RGB value of each Gaussian point is its color characteristic. Semantic features extracted by pre-trained deep learning models .

[0016] By using the built-in sensors of the user's video viewing terminal device, the system collects real-time data on the user's head and the device's motion posture, forming a sequence of user head motion trajectory features. ;

[0017] Define adjacent frame point pairs Composite similarity between The composite similarity function simultaneously measures geometric distance, appearance similarity, and semantic consistency:

[0018] Define adjacent frame point pairs Composite similarity between The composite similarity function simultaneously measures geometric distance, appearance similarity, and semantic consistency:

[0019]

[0020] In the formula, Let these be the coordinates of the Gaussian point. The negative L2 norm of the geometric distance. For color vectors, For semantic vectors, and These are the cosine similarities of the color vector and the semantic vector, respectively. These are the corresponding balance weight coefficients;

[0021] Then, the similarity within the local neighborhood is normalized using the Softmax function to obtain the probability matching matrix. ,

[0022]

[0023] In the formula, Indicates the relationship with the first The set of all points adjacent to the i-th point, i.e. the set of all points adjacent to the i-th point. The set of neighbors of each point It is a traversal The index variables of each neighbor in the probability matching matrix represent the points. Transfer to point The probability of;

[0024] Finally, the matching matrix is ​​used to perform weighted aggregation of the features of the next frame, and the points in the current frame are calculated. Aligned virtual next frame features :

[0025] .

[0026] Furthermore, the distribution model establishment method in step S200 of the present invention includes:

[0027] Quantify the degree of change of each point over time and define event scores. :

[0028]

[0029] In the formula, For geometric position balance weighting coefficients, These are the corresponding balance weight coefficients. Let these be the coordinates of the Gaussian point. For color vectors, It is a semantic vector;

[0030] The statistical distribution of event ratings was fitted using a Gaussian mixture model (GMM).

[0031]

[0032] In the formula, The mixed weight coefficients, representing static and dynamic distributions respectively, satisfy... , Represents the Gaussian probability density function; These are the mean and standard deviation of the static distribution, respectively. These are the mean and standard deviation of the dynamic distribution, respectively.

[0033] Furthermore, the optimal dynamic sensing threshold solution method in step S300 of the present invention includes:

[0034] Using the Expectation-Maximization (EM) algorithm to analyze the parameters of a Gaussian Mixture Model (GMM) Perform iterative estimation;

[0035] Closed-loop iteration of parameters is achieved by repeatedly executing the expectation step and the maximization step:

[0036] In the In the expected step of the next iteration, the event score is calculated. Belongs to static components posterior probability :

[0037]

[0038] in, Rate the event Belongs to dynamic components The posterior probability;

[0039] In the maximization step, the parameters are updated with weights based on the responsiveness:

[0040]

[0041] in Repeat the above expectation step and maximization step until the following condition is met:

[0042]

[0043] In the formula, Let be the log-likelihood function. To achieve the convergence threshold, the optimal parameter set that can accurately deconstruct the dynamic and static statistical characteristics of the scene is obtained by satisfying the above formula. ;

[0044] Secondly, based on Bayesian decision theory, we need to find the optimal split point between the two Gaussian distributions, which involves solving the roots of the following quadratic equation.

[0045]

[0046] In the formula, the coefficients of the quadratic term, the linear term, and the constant term are respectively:

[0047]

[0048]

[0049]

[0050] The solution obtained This is the optimal dynamic perception threshold, providing a basis for subsequent adaptive screening steps.

[0051] Furthermore, the adaptive screening step of the present invention includes:

[0052] If a certain point's event rating If the condition is met, the point is determined to be a dynamic salient point and input into the subsequent network for state updates; otherwise, it is determined to be a static background point and the state features of the previous frame are directly reused.

[0053] Furthermore, the method for constructing a multi-scale temporal state-space model (SSM) in step S400 of this invention includes:

[0054] Input the selected dynamic point sequence into the... Layered SSM network;

[0055] The hierarchical SSM network captures instantaneous actions and long-term trajectories, with different temporal receptive fields configured for the bottom and top layers of the network; the state update of each layer follows the following discretized state-space equation:

[0056]

[0057] In the formula, Here is the state transition matrix. For the input matrix, a gated fusion mechanism is introduced to adaptively aggregate features from different time scales:

[0058]

[0059] In the formula, The gate weights are dynamically generated for the current input features. They represent the times respectively. By the , The hidden feature vector obtained by extracting from -1 time-scale branches, To aggregate spatiotemporal features, This indicates that at time t+1, the first... The hidden feature vectors are extracted from the time-scale branches.

[0060] Furthermore, the viewpoint prediction method in step S500 of the present invention includes:

[0061] Aggregate spatiotemporal features Features of user's historical head movement trajectory data Linear concatenation is performed along the feature dimension to construct a fused feature vector. :

[0062]

[0063] In the formula, This represents a linear concatenation operation along the feature dimensions; combined with a multi-layer fully connected regression network to fuse features. The prediction process for nonlinear mapping can be expressed as follows:

[0064]

[0065] in, For fully connected layer regression networks, These are the network's learnable parameters;

[0066] And output the user viewport pose at the future target time through regression prediction. It includes quaternions of Euler angles and view cone angles of the viewport center coordinates, which determine the field of view and enable viewpoint prediction.

[0067] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0068] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0069] As can be seen from the above technical solution, the video viewpoint prediction method and system based on event-driven temporal modeling and multi-scale of the present invention aims to overcome the computational redundancy caused by the time-varying nature of point cloud topology, missing correspondences, and high data sparsity in 4DGS videos. By calculating the temporal event score of Gaussian points and using GMM adaptively to filter dynamic salient points, static redundant data is eliminated. Furthermore, multi-scale SSM is combined to capture short-term local motion and long-term temporal dependence, and user head motion trajectory features are fused. Thus, while ensuring real-time performance, efficient and accurate prediction of user viewpoint direction in immersive media scenes is achieved. Attached Figure Description

[0070] Figure 1 This is a flowchart of the video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0072] like Figure 1 As shown in this embodiment, the video viewpoint prediction method based on event-driven temporal modeling and multi-scale includes the following steps:

[0073] S100: Acquire 4DGS video viewpoint frame data to construct the current frame point. Aligned virtual next frame features;

[0074] S200, based on the current frame point Aligned virtual next-frame features are used to calculate temporal event scores for Gaussian points and establish a distribution model;

[0075] S300. The distribution model is iteratively estimated using the expectation-maximization algorithm to solve for the optimal dynamic sensing threshold and perform adaptive screening.

[0076] S400, based on adaptively selected dynamic points, constructs a multi-scale time-domain state-space model SSM;

[0077] S500, based on the multi-scale temporal state-space model SSM, completes viewpoint prediction output.

[0078] The following provides a detailed explanation of each step:

[0079] S100: Acquire 4DGS video viewpoint frame data to construct the current frame point. Aligned virtual next frame features;

[0080] Extract adjacent frames (current frame) With the next frame Spatial characteristics of Gaussian points;

[0081] Spatial features include the Gaussian point positions, i.e., geometric coordinates, in the 4DGS video viewpoint frame data. The RGB value of each Gaussian point is its color characteristic. Semantic features extracted by pre-trained deep learning models .

[0082] Simultaneously, the built-in sensors of the user's video viewing terminal device (VR headset) collect real-time motion posture data of the user's head or the device, forming a sequence of user head motion trajectory features. .

[0083] To address the issue of missing one-to-one correspondence caused by the variable number of Gaussian points and non-rigid drift in their positions in 4DGS videos, a correspondence-aware cross-frame alignment is performed.

[0084] First, in the definition of adjacent frame point pairs Composite similarity between The composite similarity function simultaneously measures geometric distance, appearance similarity, and semantic consistency:

[0085] Define adjacent frame point pairs Composite similarity between The composite similarity function simultaneously measures geometric distance, appearance similarity, and semantic consistency:

[0086]

[0087] In the formula, Let these be the coordinates of the Gaussian point. The negative L2 norm of the geometric distance. Let be a color vector, where Let i be the color vector at time t. Let j be the color vector at time t+1. Let be a semantic vector, where Let i be the semantic vector at time t. Let j be the semantic vector at time t+1. and These are the cosine similarities of the color vector and the semantic vector, respectively. These are the corresponding balance weight coefficients;

[0088] Then, the similarity within the local neighborhood is normalized using the Softmax function to obtain the probability matching matrix. ,

[0089]

[0090] In the formula, Indicates the relationship with the first The set of all points adjacent to the i-th point, i.e. the set of all points adjacent to the i-th point. The set of neighbors of each point It is a traversal The index variables of each neighbor in the probability matching matrix represent the points. Transfer to point The probability of;

[0091] Finally, the matching matrix is ​​used to perform weighted aggregation of the features of the next frame, and the points in the current frame are calculated. Aligned virtual next frame features :

[0092] .

[0093] S200, based on the current frame point Aligned virtual next-frame features are used to calculate temporal event scores for Gaussian points and establish a distribution model;

[0094] For each Gaussian point at the current moment, firstly, its position parameters, color parameters, and semantic features in the current frame and the aligned next frame are obtained; then, the position change, color change, and semantic change of the Gaussian point in adjacent moments are calculated respectively; finally, the position change, color change, and semantic change are weighted and summed according to preset weights to obtain the temporal event score of the Gaussian point. The above calculation is repeated for all Gaussian points in the current frame to obtain the set of temporal event scores corresponding to all Gaussian points, which serves as the input for subsequent distribution fitting and dynamic point selection.

[0095] To quantify the drastic change of each point over time, an event score is defined. :

[0096]

[0097] In the formula, The geometric position balance weighting coefficient, Let these be the coordinates of the Gaussian point. Let i represent the color vectors at time t and t+1, respectively. Let represent the semantic vectors of point i at time t and time t+1, respectively. Then, the event rating set for all points in the video frame is collected. Considering that a scene typically consists of a static background and a moving foreground, a Gaussian mixture model (GMM) is used to fit the statistical distribution of the event ratings:

[0098]

[0099] In the formula, E represents the score of a single event. The mixed weight coefficients, representing static and dynamic distributions respectively, satisfy... , Represents the Gaussian probability density function; These are the mean and standard deviation of the static distribution, respectively. These are the mean and standard deviation of the dynamic distribution, respectively.

[0100] S300. The distribution model is iteratively estimated using the expectation-maximization algorithm to solve for the optimal dynamic sensing threshold and perform adaptive screening.

[0101] First, the Expectation-Maximization (EM) algorithm is used to optimize the parameters of the Gaussian Mixture Model (GMM) in step 2. Perform iterative estimation.

[0102] This invention achieves closed-loop iteration of parameters by repeatedly executing the expectation step (E-step) and the maximization step (M-step):

[0103] In the In the expected step of the next iteration, the event score is calculated. Belongs to static components posterior probability :

[0104]

[0105] in, Rate the event Belongs to dynamic components The posterior probability;

[0106] In the maximization step, the parameters are updated with weights based on the responsiveness:

[0107]

[0108] in Repeat the above expectation step and maximization step until the following condition is met:

[0109]

[0110] In the formula, Let be the log-likelihood function. To achieve the convergence threshold, the optimal parameter set that can accurately deconstruct the dynamic and static statistical characteristics of the scene is obtained by satisfying the above formula. ;

[0111] Secondly, based on Bayesian decision theory, we need to find the optimal split point between the two Gaussian distributions, which involves solving the roots of the following quadratic equation.

[0112]

[0113] In the formula, the coefficients of the quadratic term, the linear term, and the constant term are respectively:

[0114]

[0115]

[0116]

[0117] The solution obtained This is the optimal dynamic sensing threshold.

[0118] If a certain point's event rating If the condition is met, the point is determined to be a dynamic salient point and input into the subsequent network for state updates; otherwise, it is determined to be a static background point and the state features of the previous frame are directly reused, thereby greatly reducing computational redundancy.

[0119] S400, based on adaptively selected dynamic points, constructs a multi-scale time-domain state-space model SSM;

[0120] Input the selected dynamic point sequence into the... A layered SSM network is used; to balance the network's ability to capture both instantaneous actions (short-term dependencies) and long-term trajectories (long-term dependencies), different temporal receptive fields are configured for the bottom and top layers. The state update of each layer follows the following discretized state-space equation:

[0121]

[0122] In the formula, Here is the state transition matrix. For the input matrix, a gated fusion mechanism is introduced to adaptively aggregate features from different time scales:

[0123]

[0124] In the formula, The gating weights are dynamically generated from the current input features. They represent the times respectively. By the , The hidden feature vector obtained by extracting from -1 time-scale branches, To aggregate spatiotemporal features, This indicates that at time t+1, the first... The hidden feature vectors extracted from each time scale branch are used to determine whether the model focuses more on local details or global trends at the current moment.

[0125] S500, based on the multi-scale temporal state-space model SSM, completes viewpoint prediction output.

[0126] The aggregated spatiotemporal features output in step S400 Features of the user's historical head movement trajectory data collected in step S100 Linear concatenation is performed along the feature dimension to construct a fused feature vector. :

[0127]

[0128] In the formula, This represents a linear concatenation operation along the feature dimensions; combined with a multi-layer fully connected regression network to fuse features. The prediction process for nonlinear mapping can be expressed as follows:

[0129]

[0130] in, For fully connected layer regression networks, The parameters are learnable by the network; and the user viewport pose at the future target time is output through regression prediction. It includes the Euler angles of the viewport center coordinates and the quaternion of the view frustum angle, thereby determining the field of view range and achieving accurate depiction of the area that the user is watching in real time in the immersive space.

[0131] In summary, this invention effectively solves the problem of missing inter-frame correlation caused by time-varying topology in 4DGS videos by constructing a correspondence-aware alignment module. Combined with a GMM-based adaptive threshold filtering mechanism, it significantly reduces computational redundancy while achieving accurate capture of dynamic salient points. Furthermore, through the fusion modeling of multi-scale SSM and user head motion features, it achieves a deep understanding of temporal evolution patterns, thereby obtaining more stable and robust viewpoint prediction results. This method ensures prediction accuracy while meeting real-time requirements, making it suitable for viewpoint prediction and related interactive tasks in 4DGS videos.

[0132] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0133] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the video viewpoint prediction methods based on event-driven temporal modeling and multi-scale described in the above embodiments.

[0134] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0135] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video viewpoint prediction method based on event-driven temporal modeling and multi-scale methods, characterized in that, Includes the following steps: S100: Acquire 4DGS video viewpoint frame data to construct the current frame point. Aligned virtual next frame features; S200, based on the current frame point Aligned virtual next-frame features are used to calculate temporal event scores for Gaussian points and establish a distribution model; S300. The distribution model is iteratively estimated using the expectation-maximization algorithm to solve for the optimal dynamic sensing threshold and perform adaptive screening. S400, based on adaptively selected dynamic points, constructs a multi-scale time-domain state-space model SSM; S500, based on the multi-scale temporal state-space model SSM, completes viewpoint prediction output; The temporal event score calculation method in step S200 includes: for each Gaussian point at the current time, firstly, obtaining the position parameters, color parameters, and semantic features of the point in the current frame and the next frame after alignment; then, calculating the position change, color change, and semantic change of the Gaussian point in adjacent time moments respectively; and then, weighting and summing the position change, color change, and semantic change according to preset weights to obtain the temporal event score of the Gaussian point.

2. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 1, characterized in that, In step S100, the current frame point is constructed. Aligned virtual next-frame feature methods include: Extract the spatial features of Gaussian points in adjacent frames, where the spatial features include the positions and geometric coordinates of Gaussian points in the 4DGS video viewpoint frame data. The RGB value of each Gaussian point is its color characteristic. Semantic features extracted by pre-trained deep learning models ; Define adjacent frame point pairs Composite similarity between The composite similarity function simultaneously measures geometric distance, appearance similarity, and semantic consistency: In the formula, Let these be the coordinates of the Gaussian point. The negative L2 norm of the geometric distance. For color vectors, For semantic vectors, and These are the cosine similarities of the color vector and the semantic vector, respectively. These are the corresponding balance weight coefficients; Then, the similarity within the local neighborhood is normalized using the Softmax function to obtain the probability matching matrix. , In the formula, Indicates the relationship with the first The set of all points adjacent to the i-th point, i.e. the set of all points adjacent to the i-th point. The set of neighbors of each point It is a traversal The index variables of each neighbor in the probability matching matrix represent the points. Transfer to point The probability of; Finally, the matching matrix is ​​used to perform weighted aggregation of the features of the next frame, and the points in the current frame are calculated. Aligned virtual next frame features : 。 3. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 1, characterized in that, The distribution model establishment method in step S200 includes: Quantify the degree of change of each point over time and define event scores. : In the formula, For geometric position balance weighting coefficients, These are the corresponding balance weight coefficients. Let these be the coordinates of the Gaussian point. For color vectors, It is a semantic vector; The statistical distribution of event ratings was fitted using a Gaussian mixture model (GMM). In the formula, E represents the score of a single event. The mixed weight coefficients, representing static and dynamic distributions respectively, satisfy... , Represents the Gaussian probability density function; These are the mean and standard deviation of the static distribution, respectively. These are the mean and standard deviation of the dynamic distribution, respectively.

4. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 1, characterized in that, The optimal dynamic sensing threshold solution method in step S300 includes: Using the Expectation-Maximization (EM) algorithm to analyze the parameters of a Gaussian Mixture Model (GMM) Perform iterative estimation; in, The mixed weight coefficients, representing static and dynamic distributions respectively, satisfy... ; These are the mean and variance of the static distribution, respectively. These are the mean and variance of the dynamic distribution, respectively. Closed-loop iteration of parameters is achieved by repeatedly executing the expectation step and the maximization step: In the In the expected step of the next iteration, the event score is calculated. Belongs to static components posterior probability : in, Rate the event Belongs to dynamic components The posterior probability; In the maximization step, the parameters are updated with weights based on the responsiveness: in Repeat the above expectation step and maximization step until the following condition is met: In the formula, Let be the log-likelihood function. To achieve the convergence threshold, the optimal parameter set that can accurately deconstruct the dynamic and static statistical characteristics of the scene is obtained by satisfying the above formula. ; Secondly, based on Bayesian decision theory, we need to find the optimal split point between the two Gaussian distributions, which involves solving the roots of the following quadratic equation. In the formula, the coefficients of the quadratic term, the linear term, and the constant term are respectively: The solution obtained This is the optimal dynamic perception threshold, providing a basis for subsequent adaptive screening steps.

5. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 4, characterized in that, The adaptive filtering step includes: If a certain point's event rating If the condition is met, the point is determined to be a dynamic salient point and input into the subsequent network for state updates; otherwise, it is determined to be a static background point and the state features of the previous frame are directly reused.

6. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 5, characterized in that, The method for constructing the multi-scale temporal state-space model (SSM) in step S400 includes: Input the selected dynamic point sequence into the... Layered SSM network; The hierarchical SSM network captures instantaneous actions and long-term trajectories, with different temporal receptive fields configured for the bottom and top layers of the network; the state update of each layer follows the following discretized state-space equation: In the formula, Here is the state transition matrix. For the input matrix, They represent the times respectively. By the , The hidden feature vectors extracted from -1 time-scale branches This indicates that at time t+1, the first... The hidden feature vectors extracted from each time scale branch are used to adaptively aggregate features from different time scales using a gated fusion mechanism. In the formula, The gate weights are dynamically generated for the current input features. It is an aggregation of spatiotemporal features.

7. The video viewpoint prediction method based on event-driven temporal modeling and multi-scale according to claim 1, characterized in that, In step S100, the current frame point is constructed. Aligned virtual next-frame feature methods also include: By using the built-in sensors of the user's video viewing terminal device, the system collects real-time data on the user's head and the device's motion posture, forming a sequence of user head motion trajectory features. ; The viewpoint prediction method in step S500 includes: Aggregate spatiotemporal features Features of user head movement trajectory data Linear concatenation is performed along the feature dimension to construct a fused feature vector. : In the formula, This represents a linear concatenation operation along the feature dimensions; combined with a multi-layer fully connected regression network to fuse features. The prediction process for nonlinear mapping can be expressed as follows: in, For fully connected layer regression networks, These are the network's learnable parameters; And output the user viewport pose at the future target time through regression prediction. It includes quaternions of Euler angles and view cone angles of the viewport center coordinates, which determine the field of view and enable viewpoint prediction.

Images