A panoramic video view angle prediction method based on a neural network

Abstract

The application discloses a panoramic video view angle prediction method based on a neural network, belongs to the technical field of image processing, and is used for interactive video processing, comprising the following steps: acquiring a panoramic video of an existing public data set, then constructing a panoramic video view angle prediction model based on structured attention to perform training, and predicting a saliency prediction view angle of the panoramic video based on the panoramic video view angle prediction model based on structured attention which has completed the training. The panoramic video view angle prediction model based on structured attention is constructed, original sparse and noisy data is converted into more robust supervision signals, the deviation of single local fixation is overcome, comprehensive modeling of scene saliency and accurate prediction of user view angle are realized, and the accuracy and robustness of view angle prediction in virtual reality application are improved.

Description

Technical Field

[0001] This invention discloses a panoramic video viewpoint prediction method based on neural networks, belonging to the field of image processing technology. Background Technology

[0002] While relying on head-mounted displays for data acquisition is feasible, it suffers from a significant drawback: human observers, limited by cognitive and physiological constraints, often cannot capture the entire 360° scene, instead focusing on localized areas. This leads to systematic biases in the data, known as observer bias. Consequently, models trained on such data are prone to learning and replicating this bias, lacking a comprehensive understanding of the panoramic scene and ultimately impacting their effectiveness in practical virtual reality applications.

[0003] In existing technologies, the closest approach is a supervised learning model based on gaze data. This involves using gaze trajectories captured by head-mounted displays (HMDs) as ground truth labels to directly train a deep learning model for viewpoint prediction. Such methods typically rely on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformer structures to model temporal features and spatial saliency. However, this approach directly depends on biased data as a supervisory signal, failing to fundamentally address the "observer bias" problem. This results in predictions that are often biased towards local areas and lack global accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a panoramic video perspective prediction method based on neural networks, in order to solve the problems in the prior art, such as over-reliance on user gaze data and the resulting observer bias, and the localization and incompleteness of model prediction results due to noise and systematic bias in the training supervision signal.

[0005] A panoramic video viewpoint prediction method based on neural networks includes:

[0006] Prepare a panoramic video dataset, construct a panoramic video view prediction model based on structured attention, train the neural network, construct an overall loss function to constrain the panoramic video view prediction model based on structured attention, output the result when the overall loss function reaches the minimum value, and generate the trained panoramic video view prediction model based on structured attention. If the overall loss function does not reach the minimum value, return to the neural network training.

[0007] The panoramic video viewpoint prediction model based on structured attention includes a parsing and construction module, a structured attention representation module, and a viewpoint generation module. The parsing and construction module includes dividing the dataset into video frame images and saliency heatmaps, inputting the data into a preprocessing layer, and generating saliency labels, image features, and adjacency matrices.

[0008] The structured attention representation module includes a graph neural network layer that receives the image features and adjacency matrix output by the parsing and construction module, processes them, and inputs them into the encoder layer of the converter. The processing result of the encoder layer of the converter is combined with the query embedding and input into the decoder layer of the converter to extract semantics. Then, it is input into a multilayer perceptron and outputs a saliency prediction score after normalization.

[0009] The viewpoint generation module includes a loss function layer that receives saliency labels and saliency prediction scores and constructs an overall loss function. The result is input into the model weight layer to determine whether the overall loss function has reached its minimum value. The module also stores and updates the training parameters of the structured attention-based panoramic video viewpoint prediction model. If the overall loss function reaches its minimum value, the corresponding saliency prediction score is input into the viewpoint generation layer. The viewpoint generation layer performs a weighted average of the spherical coordinates of the saliency prediction scores to generate the trained structured attention-based panoramic video viewpoint prediction model and outputs the saliency prediction viewpoint of the panoramic video.

[0010] The dataset consists of panoramic videos. The data preprocessing layer of the parsing and construction module adopts a multi-view stereo projection strategy, projecting each frame of video onto 6 standard two-dimensional cube faces. On each stereo face, a pre-trained two-dimensional object detection model is used to generate two-dimensional candidate boxes as candidate box features of the model.

[0011] The data preprocessing layer constructs an undirected graph for each frame of panoramic video image. , For a set of nodes, Let be the set of edges. For nodes, Corresponding The number of candidate objects is , The initial feature vector is a concatenation of visual features and initial saliency scores. initial feature vector for:

[0012] ;

[0013] In the formula, For vector concatenation, Visual features are obtained by feature pooling in the candidate bounding box region on the face of a two-dimensional cube; The initial significance score is . Normalized result of the number of line-of-sight points within the boundary; For node indexing, ; The formula is:

[0014] ;

[0015] In the formula, The total number of viewpoints. To find the maximum value, Indexing the line-of-sight points within the molecule. , For the index of the line-of-sight point in the denominator, , This is an indicator function.

[0016] Set distance threshold Intersection and comparison threshold If the distance between two candidate boxes in the spherical coordinate system is less than a threshold , or bounding box Greater than Establish an edge between the two candidate boxes. :

[0017] ;

[0018] In the formula, The distance is spherical. The longitude is shown on the panoramic video unfolded map. The latitude and longitude on the panoramic video unfolded map. for longitude, for latitude, for longitude, for latitude, for The second index, , For intersection, union, and comparison.

[0019] Graph neural network layers include, and The input is fed into the graph neural network layer, and This is concretized into an initial adjacency matrix. ,Will This is concretized into an initial feature matrix. The update rule for the graph neural network layer is:

[0020] ;

[0021] In the formula, For the first The output feature matrix of the layer, This is the layer index of a graph neural network. , The number of layers in a graph neural network. To add a self-loop to the adjacency matrix, , It is the identity matrix. for The corresponding degree matrix, For learnable weights, For activation functions;

[0022] The output of the graph neural network layer is the candidate box feature enhanced with local context. :

[0023] = ;

[0024] In the formula, For the first The final output feature vector of each node.

[0025] Will Dimensional transformation is performed on the input linear layer using positional encoding. ,for Inject position signal, obtain Initial representation vector input to the encoder :

[0026] ;

[0027] In the formula, Linear layer;

[0028] Will The input converter's encoder layer iteratively integrates global semantic information through multi-head self-attention.

[0029] The decoder layer of the converter uses query embedding. As input, The learning vector is used to generate the first... The target sequence at each position The index of the target sequence. The decoder layer of the converter extracts semantics from the global representation output by the encoder layer through a multi-head cross-attention mechanism:

[0030] ;

[0031] In the formula, For the first One significant inference result, This is the final output after processing by the encoder layer. For the first The final output after each node has been processed by the encoder layer For normalization, It is the transpose symbol. In the attention mechanism Vector dimension; handling all One query , get all Significant inference results .

[0032] Will The input is a multilayer perceptron, which consists of two feedforward networks. The first feedforward network includes a linear layer. and Activation function, the second layer of the feedforward network includes linear layers and Activation function;

[0033] definition The output dimension will enter Then through The activation function introduces non-linearity, and then through Will The activated features are aggregated into a single scalar value, and then the single scalar value is passed through... Activation function normalization:

[0034] ;

[0035] In the formula, The saliency prediction score is the score for the candidate box.

[0036] The loss function layer includes the use of binary cross-entropy loss and... The combination of losses constructs the overall loss function :

[0037] ;

[0038] ;

[0039] ;

[0040] In the formula, For standard binary cross-entropy loss, for loss, For all A set of significance prediction scores for each prediction. To predict the significance score for matching the true value, For all A set of significance prediction scores matching true values. To weigh parameters;

[0041] The model weight layer includes judgment. Whether it is the minimum value, and store and update the training parameters of the panoramic video viewpoint prediction model based on structured attention; if When the minimum value is reached, the corresponding Output; if If the minimum value is not reached, return to neural network training.

[0042] The viewpoint generation layer includes aggregating saliency prediction scores, identifying the center points of candidate boxes with saliency prediction scores higher than the threshold between saliency and non-saliency objects as attention centers, and performing a weighted average of the spherical coordinates of the attention centers to obtain the saliency prediction viewpoint of the panoramic video.

[0043] ;

[0044] In the formula, To predict the longitude of the viewpoint for saliency in panoramic video, The latitude of the viewpoint is used to predict the saliency of panoramic video. For the first The longitude of the center of attention, For the first The latitude of the center of attention.

[0045] Compared with existing technologies, the present invention has the following advantages: By avoiding direct reliance on biased gaze data and introducing intermediate structured attention representations, the present invention transforms the original sparse and noisy data into a more robust supervisory signal; by fusing local and global information and using graph neural networks to analyze the spatial relationships between candidate regions, and combining a converter to perform global contextual reasoning, the present invention overcomes the bias of single local gaze; it achieves comprehensive modeling of scene salience and accurate prediction of user perspective, improves the accuracy and robustness of perspective prediction in virtual reality applications, and thus optimizes the user experience of applications such as foveated rendering, viewport adaptive transmission, and attention-based interaction. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the network structure of the panoramic video view prediction model based on structured attention in this invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] A panoramic video viewpoint prediction method based on neural networks includes:

[0049] Prepare a panoramic video dataset, construct a panoramic video view prediction model based on structured attention, train the neural network, construct an overall loss function to constrain the panoramic video view prediction model based on structured attention, output the result when the overall loss function reaches the minimum value, and generate the trained panoramic video view prediction model based on structured attention. If the overall loss function does not reach the minimum value, return to the neural network training.

[0050] The panoramic video viewpoint prediction model based on structured attention includes a parsing and construction module, a structured attention representation module, and a viewpoint generation module. The parsing and construction module includes dividing the dataset into video frame images and saliency heatmaps, inputting the data into a preprocessing layer, and generating saliency labels, image features, and adjacency matrices.

[0051] The structured attention representation module includes a graph neural network layer that receives the image features and adjacency matrix output by the parsing and construction module, processes them, and inputs them into the encoder layer of the converter. The processing result of the encoder layer of the converter is combined with the query embedding and input into the decoder layer of the converter to extract semantics. Then, it is input into a multilayer perceptron and outputs a saliency prediction score after normalization.

[0052] The viewpoint generation module includes a loss function layer that receives saliency labels and saliency prediction scores and constructs an overall loss function. The result is input into the model weight layer to determine whether the overall loss function has reached its minimum value. The module also stores and updates the training parameters of the structured attention-based panoramic video viewpoint prediction model. If the overall loss function reaches its minimum value, the corresponding saliency prediction score is input into the viewpoint generation layer. The viewpoint generation layer performs a weighted average of the spherical coordinates of the saliency prediction scores to generate the trained structured attention-based panoramic video viewpoint prediction model and outputs the saliency prediction viewpoint of the panoramic video.

[0053] The dataset consists of panoramic videos. The data preprocessing layer of the parsing and construction module adopts a multi-view stereo projection strategy, projecting each frame of video onto 6 standard two-dimensional cube faces. On each stereo face, a pre-trained two-dimensional object detection model is used to generate two-dimensional candidate boxes as candidate box features of the model.

[0054] The data preprocessing layer constructs an undirected graph for each frame of panoramic video image. , For a set of nodes, Let be the set of edges. For nodes, Corresponding The number of candidate objects is , The initial feature vector is a concatenation of visual features and initial saliency scores. initial feature vector for:

[0055] ;

[0056] In the formula, For vector concatenation, Visual features are obtained by feature pooling in the candidate bounding box region on the face of a two-dimensional cube; The initial significance score is . Normalized result of the number of line-of-sight points within the boundary; For node indexing, ; The formula is:

[0057] ;

[0058] In the formula, The total number of viewpoints. To find the maximum value, Indexing the line-of-sight points within the molecule. , For the index of the line-of-sight point in the denominator, , This is an indicator function.

[0059] Set distance threshold Intersection and comparison threshold If the distance between two candidate boxes in the spherical coordinate system is less than a threshold , or bounding box Greater than Establish an edge between the two candidate boxes. :

[0060] ;

[0061] In the formula, The distance is spherical. The longitude is shown on the panoramic video unfolded map. The latitude and longitude on the panoramic video unfolded map. for longitude, for latitude, for longitude, for latitude, for The second index, , For intersection, union, and comparison.

[0062] Graph neural network layers include, and The input is fed into the graph neural network layer, and This is concretized into an initial adjacency matrix. ,Will This is concretized into an initial feature matrix. The update rule for the graph neural network layer is:

[0063] ;

[0064] In the formula, For the first The output feature matrix of the layer, This is the layer index of a graph neural network. , The number of layers in a graph neural network. To add a self-loop to the adjacency matrix, , It is the identity matrix. for The corresponding degree matrix, For learnable weights, For activation functions;

[0065] The output of the graph neural network layer is the candidate box feature enhanced with local context. :

[0066] = ;

[0067] In the formula, For the first The final output feature vector of each node.

[0068] Will Dimensional transformation is performed on the input linear layer using positional encoding. ,for Inject position signal, obtain Initial representation vector input to the encoder :

[0069] ;

[0070] In the formula, Linear layer;

[0071] Will The input converter's encoder layer iteratively integrates global semantic information through multi-head self-attention.

[0072] The decoder layer of the converter uses query embedding. As input, The learning vector is used to generate the first... The target sequence at each position The index of the target sequence. The decoder layer of the converter extracts semantics from the global representation output by the encoder layer through a multi-head cross-attention mechanism:

[0073] ;

[0074] In the formula, For the first One significant inference result, This is the final output after processing by the encoder layer. For the first The final output after each node has been processed by the encoder layer For normalization, It is the transpose symbol. In the attention mechanism Vector dimension; handling all One query , get all Significant inference results .

[0075] Will The input is a multilayer perceptron, which consists of two feedforward networks. The first feedforward network includes a linear layer. and Activation function, the second layer of the feedforward network includes linear layers and Activation function;

[0076] definition The output dimension will enter Then through The activation function introduces non-linearity, and then through Will The activated features are aggregated into a single scalar value, and then the single scalar value is passed through... Activation function normalization:

[0077] ;

[0078] In the formula, The saliency prediction score is the score for the candidate box.

[0079] The loss function layer includes the use of binary cross-entropy loss and... The combination of losses constructs the overall loss function :

[0080] ;

[0081] ;

[0082] ;

[0083] In the formula, For standard binary cross-entropy loss, for loss, For all A set of significance prediction scores for each prediction. To predict the significance score for matching the true value, For all A set of significance prediction scores matching true values. To weigh the parameters;

[0084] The model weight layer includes judgment. Whether it is the minimum value, and store and update the training parameters of the panoramic video viewpoint prediction model based on structured attention; if When the minimum value is reached, the corresponding Output; if If the minimum value is not reached, return to neural network training.

[0085] The viewpoint generation layer includes aggregating saliency prediction scores, identifying the center points of candidate boxes with saliency prediction scores higher than the threshold between saliency and non-saliency objects as attention centers, and performing a weighted average of the spherical coordinates of the attention centers to obtain the saliency prediction viewpoint of the panoramic video.

[0086] ;

[0087] In the formula, To predict the longitude of the viewpoint for saliency in panoramic video, The latitude of the viewpoint is used to predict the saliency of panoramic video. For the first The longitude of the center of attention, For the first The latitude of the center of attention.

[0088] The following explanation, in conjunction with the accompanying drawings, will provide further details. Figure 1As shown, the panoramic video viewpoint prediction model based on structured attention includes a parsing and construction module, a structured attention representation module, and a viewpoint generation module. The parsing and construction module divides the dataset into video frame images and saliency heatmaps, inputs them into a data preprocessing layer, and generates saliency labels, image features, and an adjacency matrix. The structured attention representation module includes a graph neural network layer that receives the image features and adjacency matrix output from the parsing and construction module, processes them, and inputs them into the encoder layer of the converter. The encoder layer's processing result, combined with query embedding, is input into the decoder layer of the converter to extract semantics, and then input into a multilayer perceptron. After normalization, it outputs a saliency prediction score. The viewpoint generation module includes a loss function layer that receives the saliency labels and saliency prediction scores and constructs an integral viewpoint generation module. The overall loss function is input into the model weight layer to determine whether the overall loss function has reached its minimum value. The training parameters of the structured attention-based panoramic video viewpoint prediction model are stored and updated. If the overall loss function reaches its minimum value, the corresponding saliency prediction score is input into the viewpoint generation layer and the model parameters are updated. The viewpoint generation layer performs a weighted average of the spherical coordinates of the saliency prediction scores to generate the trained structured attention-based panoramic video viewpoint prediction model and outputs the saliency prediction viewpoint of the panoramic video. The pre-trained two-dimensional object detection models include CIINet (Cross-modal Interaction and Integration Network), VSCode (Visual-Semantic Encoder), VSTNet (Visual Transformer Network), and SSFam (Single-Stage Detection Family Model). This represents the maximum radius in radians between the two targets in the spherical coordinate system. This represents the lower limit of the spatial overlap between two candidate boxes.

[0089] This invention employs four quantitative evaluation metrics. Compared with other existing technologies, these include: S-measure (Sm): used to evaluate the structural similarity between the predicted saliency map and the ground truth annotation at both the region and target perception levels; F-measure (maxF): considering both precision and recall, the maximum value of this metric across all test images in this invention; E-measure (maxE): considering both pixel-level and image-level errors, the maximum value of this metric across all test images in this invention; and Mean Absolute Error (MAE): calculating the average absolute error between the predicted saliency map and the ground truth annotation at the pixel level.

[0090] The dataset is the 360SOD dataset. Set to 0.15, The threshold for the boundary between salient and non-salient objects was set to 0.1, and 0.7. Nine representative salient object detection (SOD) models were selected and divided into two main categories: The first category consists of 3D object detection models, including five advanced methods designed specifically for panoramic images: DDS (Depth Distribution Sensitive Network), FANet (Feature Alignment Network), LDNet (Local Denoising Network), and MRFRNet (Multi-Scale Receptive Field Refinement Network). The network structures of these methods are specifically designed for the characteristics of spherical images. The second category consists of high-performance 2D object detection models, including MINet (Multi-Interaction Network), GeleNet (Global Extraction Local Exploration Network), ADMNet (Attention-Driven Multi-Scale Network), ColA (Context and Local Attention), and NASAL (Neural Structure Search Reinforcement Learning). These methods were not designed for panoramic data but were used to test the performance of the proposed framework compared to powerful non-specialized solutions; as shown in Table 1.

[0091] Table 1. Comparison results of the present invention with other methods on the 360SOD dataset.

[0092] ;

[0093] Comparative analysis shows that the model of this invention significantly outperforms all 3D object detection models specifically designed for panoramic data. Compared to the best-performing baseline method MPFRNet, the method of this invention achieves a 0.51% improvement in S-measure (0.8417 vs. 0.8460), a 2.67% improvement in maxF-measure (0.765 vs. 0.786), and a 2.85% improvement in maxE-measure (0.885 vs. 0.911), while reducing the mean absolute error (MAE) by 3.14% (0.0191 vs. 0.0185). This significant performance gap strongly validates the core argument of this invention: the strategy of robustly estimating the optimal viewpoint first and then performing fine-grained analysis on that local view is more effective than end-to-end methods that directly process the entire geometrically distorted panoramic image.

[0094] Furthermore, this invention also conducted user research to further verify the subjective impact of the proposed viewpoint prediction framework on user experience in a virtual reality environment. High-quality viewpoint prediction can significantly improve the effects of gaze rendering and adaptive streaming, thereby improving users' perception of visual quality, latency, and immersion. To this end, this invention recruited 50 participants aged 20 to 35, half male and half female, with normal or corrected vision and experience using virtual reality devices. Each participant watched the same 10 panoramic videos under three viewpoint prediction conditions: a traditional regression method based on historical head-mounted display gaze (BL1), a general method based on 360° saliency prediction (BL2), and the method of this invention (SAR-Pred). During the experiment, the video order and conditions were randomized and balanced to avoid order effects. After the viewing session, participants were required to complete a questionnaire. The questionnaire focused on five key subjective indicators directly affected by the accuracy and robustness of perspective prediction, including: Perceived Clarity (focusing on whether the main area is clear and sharp); Visual Continuity (whether the viewing process is smooth without obvious jumps); Immersion (whether one can be fully immersed without being disturbed by artifacts or blur); Comfort (whether the viewing experience is natural and comfortable, without causing fatigue or dizziness); and Responsiveness (whether the system can promptly present the high-quality content expected by the user). The results are shown in Table 2.

[0095] Table 2. User Survey Results

[0096] ;

[0097] As shown in Table 2, the method of this invention significantly outperformed the two baseline methods on all evaluated subjective perception metrics (all metrics p < 0.05, most metrics p < 0.001). Specifically, the method of this invention showed significant improvements in perceptual clarity and visual continuity, indicating that the area of ​​focus for the user remained clear and sharp during head movement during fixation rendering. Improved scores for immersion and comfort indicate reduced cognitive load and enhanced presence. Most importantly, the significant improvement in responsiveness metrics (p < 0.001) directly validated the hypothesis of this invention: mitigating observer bias through structured attention representation enables efficient prediction of user intent and future perspective. These subjective evaluation results, combined with the aforementioned quantitative metrics, fully demonstrate that the method of this invention can translate more accurate perspective prediction into a significantly superior user experience in virtual reality environments.

[0098] This invention addresses the core challenge in panoramic view prediction: the pervasive observer bias in gaze supervision data from head-mounted displays. It points out that this bias stems from the cognitive and physiological limitations of humans exploring 360° scenes, leading to systematic defects in gaze data annotation. Consequently, traditional models can only learn localized scene salience. To address this, this invention proposes a novel framework based on structured attention representation, acting as a robust intermediate surrogate to mitigate bias and infer true saliency. Experimental results validate the superior performance of the proposed method in diverse panoramic environments. By decoupling the learning of attention representation from noisy gaze signals, this invention provides a more reliable paradigm for VR attention modeling, offering a new solution for user view prediction in immersive media.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panoramic video viewpoint prediction method based on neural networks, characterized in that, include: Prepare a panoramic video dataset, construct a panoramic video view prediction model based on structured attention, train the neural network, construct an overall loss function to constrain the panoramic video view prediction model based on structured attention, output the result when the overall loss function reaches the minimum value, and generate the trained panoramic video view prediction model based on structured attention. If the overall loss function does not reach the minimum value, return to train the neural network. The panoramic video viewpoint prediction model based on structured attention includes a parsing and construction module, a structured attention representation module, and a viewpoint generation module. The parsing and construction module includes dividing the dataset into video frame images and saliency heatmaps, inputting the data into a preprocessing layer, and generating saliency labels, image features, and adjacency matrices. The structured attention representation module includes a graph neural network layer that receives the image features and adjacency matrix output by the parsing and construction module, processes them, and inputs them into the encoder layer of the converter. The processing result of the encoder layer of the converter is combined with the query embedding and input into the decoder layer of the converter to extract semantics. Then, it is input into a multilayer perceptron and outputs a saliency prediction score after normalization. Viewpoint generation module The process includes a loss function layer that receives saliency labels and saliency prediction scores and constructs an overall loss function. The result is input into the model weight layer to determine whether the overall loss function has reached its minimum value. The training parameters of the structured attention-based panoramic video viewpoint prediction model are stored and updated. If the overall loss function reaches its minimum value, the corresponding saliency prediction score is input into the viewpoint generation layer. The viewpoint generation layer performs a weighted average of the spherical coordinates of the saliency prediction scores to generate the trained structured attention-based panoramic video viewpoint prediction model and outputs the saliency prediction viewpoint of the panoramic video. The dataset consists of panoramic videos. The data preprocessing layer of the parsing and construction module adopts a multi-view stereo projection strategy, projecting each frame of video onto 6 standard two-dimensional cube faces. On each stereo face, a pre-trained two-dimensional object detection model is used to generate two-dimensional candidate boxes as candidate box features of the model. The data preprocessing layer constructs an undirected graph for each frame of panoramic video image. , For a set of nodes, Let be the set of edges. For nodes, Corresponding The number of candidate objects is , The initial feature vector is a concatenation of visual features and initial saliency scores. initial feature vector for: ； In the formula, For vector concatenation, Visual features are obtained by feature pooling in the candidate bounding box region on the face of a two-dimensional cube; The initial significance score is . Normalized result of the number of line-of-sight points within the boundary; For node indexing, ; The formula is: ； In the formula, The total number of viewpoints. To find the maximum value, Indexing the line-of-sight points within the molecule. , For the index of the line-of-sight point in the denominator, , This is an indicator function.

2. The panoramic video viewpoint prediction method based on neural networks as described in claim 1, characterized in that, Set distance threshold Intersection and comparison threshold If the distance between two candidate boxes in the spherical coordinate system is less than a threshold , or bounding box Greater than Establish an edge between the two candidate boxes. : ； In the formula, The distance is spherical. The longitude is shown on the panoramic video unfolded map. The latitude and longitude on the panoramic video unfolded map. for longitude, for latitude, for longitude, for latitude, for The second index, , For intersection, union, and comparison.

3. The panoramic video viewpoint prediction method based on neural networks as described in claim 2, characterized in that, Graph neural network layers include, and The input is fed into the graph neural network layer, and This is concretized into an initial adjacency matrix. ,Will This is concretized into an initial feature matrix. The update rule for the graph neural network layer is: ； In the formula, For the first The output feature matrix of the layer, This is the layer index of a graph neural network. , The number of layers in a graph neural network. To add a self-loop to the adjacency matrix, , It is the identity matrix. for The corresponding degree matrix, For learnable weights, For activation functions; The output of the graph neural network layer is the candidate box feature enhanced with local context. : = ； In the formula, For the first The final output feature vector of each node.

4. The panoramic video viewpoint prediction method based on neural networks as described in claim 3, characterized in that, Will Dimensional transformation is performed on the input linear layer using positional encoding. ,for Inject position signal, obtain Initial representation vector input to the encoder : ； In the formula, Linear layer; Will The input converter's encoder layer iteratively integrates global semantic information through multi-head self-attention.

5. The panoramic video viewpoint prediction method based on neural networks as described in claim 4, characterized in that, The decoder layer of the converter uses query embedding. As input, The learning vector is used to generate the first... The target sequence at each position The index of the target sequence. The decoder layer of the converter extracts semantics from the global representation output by the encoder layer through a multi-head cross-attention mechanism: ； In the formula, For the first One significant inference result, This is the final output after processing by the encoder layer. For the first The final output after each node has been processed by the encoder layer For normalization, It is the transpose symbol. In the attention mechanism Vector dimension; handling all One query , get all Significant inference results .

6. The panoramic video viewpoint prediction method based on neural networks as described in claim 5, characterized in that, Will The input is a multilayer perceptron, which consists of two feedforward networks. The first feedforward network includes a linear layer. and Activation function, the second layer of the feedforward network includes linear layers and Activation function; definition The output dimension will enter Then through The activation function introduces non-linearity, and then through Will The activated features are aggregated into a single scalar value, and then the single scalar value is passed through... Activation function normalization: ； In the formula, The saliency prediction score is the score for the candidate box.

7. The panoramic video viewpoint prediction method based on neural networks as described in claim 6, characterized in that, The loss function layer includes the use of binary cross-entropy loss and... The combination of losses constructs the overall loss function : ； ； ； In the formula, For standard binary cross-entropy loss, for loss, For all A set of significance prediction scores for each prediction. To predict the significance score for matching the true value, For all A set of significance prediction scores for matching true values. To weigh parameters; The model weight layer includes judgment. Whether it is the minimum value, and store and update the training parameters of the panoramic video viewpoint prediction model based on structured attention; if When the minimum value is reached, the corresponding Output; if If the minimum value is not reached, return to neural network training.

8. The panoramic video viewpoint prediction method based on neural networks as described in claim 7, characterized in that, The viewpoint generation layer includes aggregating saliency prediction scores, identifying the center points of candidate boxes with saliency prediction scores higher than the threshold between saliency and non-saliency objects as attention centers, and performing a weighted average of the spherical coordinates of the attention centers to obtain the saliency prediction viewpoint of the panoramic video. ； In the formula, To predict the longitude of the viewpoint for saliency in panoramic video, The latitude of the viewpoint is used to predict the saliency of panoramic video. For the first The longitude of the center of attention, For the first The latitude of the center of attention.

Images