BiLSTM-based 360-degree video edge auxiliary viewport prediction method and system

Abstract

The invention provides a BiLSTM-based 360-degree video edge auxiliary viewport prediction method and system, and the method comprises the steps: distributing a video into a plurality of segments, carrying out the dividing of the segments, carrying out the projection, carrying out the detection of an expansion image obtained through projection, and tracking the motion track of a target object through a result; the client sends the historical viewport information of the user to the edge server, and the edge server inputs the historical viewport information of the user and the target movement track information into the BiLSTM prediction model to obtain future viewport coordinates; performing bit rate distribution according to the obtained future viewport coordinates, performing bit rate assignment on the video division part through a designed bit rate decision and receiving network broadband information, and caching and rendering video data requested by a cloud server by an edge server; and calculating a user experience quality index through the quality of the video watched by the user. According to the invention, the bandwidth occupation of a non-key area is reduced through an adaptive bit rate allocation scheme, and the bandwidth waste is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer video processing, and in particular to a 360° video edge-assisted viewport prediction method and system based on BiLSTM. Background Art

[0002] When users watch 360° videos, they can freely drag or rotate the viewing angle to fully explore the scene content in the video, which greatly enhances the immersiveness and realism of the viewing experience. The projection area within the user's line of sight is defined as the field of view.

[0003] However, there are some challenges in 360° video applications, making it difficult to achieve the viewing experience that users expect. On the one hand, 360° video has a huge demand for network bandwidth due to its ultra-high resolution, and this demand is the key to ensuring that users can enjoy a high-quality viewing experience. Compared with traditional 2D video, 360° video has a significantly increased demand for bandwidth, and the bandwidth required is usually 4 to 5 times that of traditional video. On the other hand, in the viewing experience of panoramic video, latency is a crucial consideration. According to research, only when the latency is less than 20ms can users be provided with a smooth and comfortable viewing experience. This requires that the transmission network must have high-fidelity and low-latency technical characteristics to ensure real-time transmission and accurate presentation of video data. Under the field of view constraints of head-mounted displays, users cannot perceive the entire content of 360° videos. In fact, they can only observe less than 20% of the content in the panoramic view at any given moment.

[0004] To address these challenges, researchers proposed a tile-based 360° video streaming method; 360° video frames are divided into multiple rectangular areas of equal size, called tiles. However, using a fixed blocking scheme, only tiles within the user's field of view are transmitted, which may cause blank screens when the user's head moves vigorously; based on the fact that viewers can only see a very small area of ​​the entire 360° video at any time, we map the 360° video into equirectangular video frames, and then divide the equirectangular video frames into 8x8 tiled tiles; considering the information collection time and the network delay between the server and the client in real network scenarios, real-time interactive streaming media systems require an accurate viewport prediction scheme to predict the user's future viewing position in advance. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a 360° video edge-assisted viewport prediction method and system based on BiLSTM to solve the above technical problems.

[0006] The present invention proposes a 360° video edge-assisted viewport prediction method and system based on BiLSTM, the method comprising the following steps: Step 1: Divide the 360° video stored in the cloud server into several segments, and project each frame of each segment to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the motion trajectory information of the target object is generated according to the coordinate sequence of the object, and the motion trajectory information of the target object is sent to the edge server; Step 2: The client sends the user's historical viewport information to the edge server. The edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result. The prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates. Step 3: Based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The edge server requests the cloud server for video data in advance for caching and rendering according to the result of bit rate assignment. Step 4: The client sends a request to the edge server to obtain predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

[0007] The present invention also proposes a 360° video edge-assisted viewport prediction system based on BiLSTM, the system comprising: Video preprocessing module for: The 360° video stored in the cloud server is divided into several segments, and each frame of each video segment is divided and projected to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the target object motion trajectory information is generated according to the object coordinate sequence, and the target object motion trajectory information is sent to the edge server; Edge-viewport prediction module, used to: The client sends the user's historical viewport information to the edge server, and the edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result, and the prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates; Edge-precaching decision module, used to: Based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The edge server requests the cloud server to cache and render the video data in advance based on the result of bit rate assignment. Client-video playback module, used for: The client sends a request to the edge server to obtain the predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

[0008] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention combines the strategies of target detection and viewport prediction to better capture the impact of video content on the user's viewport trajectory, thereby achieving more accurate and efficient viewport prediction and improving user experience; 2. The present invention designs a new BiLSTM network for viewport prediction, so that the BiLSTM model can efficiently capture and utilize time dependency, and cooperates with the introduced target detection module to provide more accurate and personalized viewport prediction results, thereby significantly improving the accuracy of viewport prediction and greatly enhancing the robustness of the algorithm; 3. The present invention solves the problems of video fluctuation and limited available wireless bandwidth by designing a 360° video adaptive streaming algorithm based on viewport prediction, while reducing redundancy between files and improving user experience.

[0009] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will become apparent from the following description, or will be learned through embodiments of the present invention. Additional aspects and advantages of the present invention will be given in part in the following description, and in part will become apparent from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flowchart of the steps of the BiLSTM-based 360° video edge-assisted viewport prediction method proposed in the present invention.

[0011] Figure 2 This is a schematic diagram of the edge network structure in the BiLSTM-based 360° video edge-assisted viewport prediction method proposed in the present invention.

[0012] Figure 3This is a schematic diagram of the minimum unit division method of a 360° video file in the BiLSTM-based 360° video edge-assisted viewport prediction method proposed in the present invention.

[0013] Figure 4 This is a schematic diagram of a viewport prediction model based on BiLSTM in the 360° video edge-assisted viewport prediction method based on BiLSTM proposed in the present invention.

[0014] Figure 5 Schematic diagram of the overall framework of the BiLSTM-based 360° video edge-assisted viewport prediction system proposed in the present invention. DETAILED DESCRIPTION

[0015] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0016] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0017] See also Figure 1 The embodiment of the present invention proposes a 360° video edge-assisted viewport prediction method based on BiLSTM, the method comprising the following steps: Step 1: Divide the 360° video stored in the cloud server into several segments, and project each frame of each segment to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the motion trajectory information of the target object is generated according to the coordinate sequence of the object, and the motion trajectory information of the target object is sent to the edge server.

[0018] See also Figure 3 In step 1, the 360° video stored in the cloud server is divided into several segments, and each frame of each segment of the video is divided and projected to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain a coordinate sequence of the object, and the motion trajectory information of the target object is generated according to the coordinate sequence of the object. Specifically, the steps include: The 360° video is segmented in the time dimension to obtain segmented video frames, and the segmented video frames are mapped on a two-dimensional plane to obtain an unfolded image; The YOLOv3 target detection algorithm is used to detect the unfolded image to obtain the detection results, the detection results are processed by boundaries and assigned numbers, and the coordinate information of the objects to which the number assignment is completed is recorded. The coordinate sequence of the objects with corresponding numbers is generated according to the recorded coordinate information; Among them, the boundary processing process corresponds to the following relationship: ; in, Indicates The horizontal coordinate of the target object, Indicates the video frame width; The coordinate sequences of the objects with corresponding numbers are connected to track the motion trajectory of the numbered objects to obtain the trajectory result, and the tracked trajectory result is sent to the edge server.

[0019] Furthermore, the grouping and slicing of equal numbers of equirectangular video frames in step 1 refers to mapping the 360° video onto a two-dimensional plane using equirectangular projection. In the time dimension, the 360° video is evenly divided into Each segment contains an equal number of consecutive frames; in the spatial dimension, each video frame is divided into 8x8 tiles, and each tile is encoded in multiple versions with different quality levels and stored in the cloud server.

[0020] In step 1, an improved object detection algorithm based on YOLOv3 is used on the video frames to transform the object detection problem into a regression problem, and the object category and bounding box are predicted simultaneously in a single neural network; the main steps are as follows: Input image processing: scaling the input image to a uniform size while maintaining the original image ratio. The image is divided into several grids, each of which is responsible for detecting the object at its center; Feature extraction: Extract multi-scale image features through convolutional neural networks. Image features help the network recognize objects of different sizes; Multi-scale prediction: YOLOv3 makes predictions at three different scales to accommodate targets of small, medium, and large sizes. This operation makes YOLOv3 more robust when dealing with complex scenes. Bounding box prediction: For each grid, the network predicts multiple bounding boxes and outputs the coordinates, width, height, confidence, and category probability of each box. The confidence indicates whether there is an object in the box and the accuracy of the prediction; Non-maximum suppression: In order to remove the bounding boxes of repeated detections, the algorithm uses non-maximum suppression technology to only retain bounding boxes with high confidence and no overlap.

[0021] Among them, due to the circumference of 360° video, special attention should be paid to the target objects at the border of the video to ensure that they can be correctly identified as the same object. Therefore, the consistency in time and space is used to identify objects. Since objects usually do not change drastically in an instant, based on this rule, the position of the object on the time axis can be tracked to determine whether it has moved from one border to the other side; During the detection process, a unique number is assigned to each object detected in the video, and then the coordinate information of each target object in each frame is recorded to form a coordinate sequence of the object; the generated coordinate sequences are connected to track the motion trajectory of each target object, and the tracked trajectory information is sent to the edge server; at the same time, the client also needs to send the user's historical viewport trajectory information, buffer size and current network bandwidth information to the edge server, and the edge server receives them together.

[0022] Furthermore, the spherical format of the 360° video is first projected into a two-dimensional video format using an equirectangular projection algorithm, and then the smallest storage unit is obtained by segmenting the 360° video from the time dimension, the space dimension, and the coding dimension. The time dimension refers to segmenting the 360° video with a total duration of a preset time every preset time, and finally obtaining video clips; The spatial dimension refers to the segmentation of each video frame. In this embodiment, the video frame after equirectangular projection is divided into 8x8 tiles. Therefore, the video screen is divided into 64 rectangular areas of equal size, each of which is called a tile, and the spatial dimension is numbered from 0 to 63; In terms of encoding dimension, this embodiment adopts a scalable encoding method to encode the video into files of three different definition levels, namely low, medium and high definition.

[0023] Step 2: The client sends the user's historical viewport information to the edge server. The edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result. The prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates.

[0024] See also Figure 4In step 2, the client sends the user's historical viewport information to the edge server, and the edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result, and the prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates, which specifically includes the following steps: The client sends the user's historical viewport trajectory information to the edge server, and the edge server pre-processes the received user's historical viewport trajectory information; Initialize the BiLSTM prediction model and send the user's historical viewport trajectory information preprocessed by the edge server to the initialized BiLSTM prediction model; The forward LSTM of the BiLSTM prediction model is processed in chronological order and input into the viewport sequence, and the backward LSTM of the BiLSTM prediction model is processed in the reverse order of chronological order and input into the viewport sequence. The forward LSTM and the backward LSTM jointly process the received information and detect the expanded image to obtain the detection result to form the prediction result.

[0025] The client sends the user's historical viewport trajectory information to the edge server, and the edge server preprocesses the received user's historical viewport trajectory information, specifically including the following steps: The maximum number of video frames is calculated based on the timestamp of the target object's motion trajectory. The corresponding process has the following relationship: ; in, Represents iteration Find the value with the largest timestamp. Indicates the number of frames per second. Indicates the timestamp, Indicates the maximum number of frames per second. Indicates converting the calculation result into an integer; The viewport trajectory data is converted to the position coordinates on the corresponding window. The corresponding process has the following relationship: ; in, express The values ​​of the axis coordinates, express The values ​​of the axis coordinates, Represents the original viewport of the current frame Axis coordinate scale information, Represents the original viewport of the current frame Axis coordinate scale information, Indicates the video frame width, Indicates the video frame height, Indicates the original width of the 360° video. Indicates the original height of the 360° video.

[0026] It should be noted that and The viewport trajectory data of the 360° video can be mapped to a specific position in the display window. Usually a relative ratio, such as a value between 0 and 1; The width is 2560 pixels, The height is 1440 pixels. These values ​​are used to convert relative scale to actual pixel coordinates; and It is to convert the original scale data to the corresponding scale of the display window.

[0027] In the step of processing the forward LSTM of the BiLSTM prediction model in chronological order and inputting it into the viewport sequence, processing the backward LSTM of the BiLSTM prediction model in the reverse order of chronological order and inputting it into the viewport sequence, and processing the received information by the forward LSTM and the backward LSTM together to detect the expanded image to obtain the detection result to form the prediction result, the corresponding relationship between the process is as follows: ; in, Indicates the current status. represents the forward output state, Indicates the backward output state, Indicates the current time step The feature vector input is represents the forward LSTM propagation along the time direction, represents the backward LSTM propagation along time, Represents the output state obtained by the forward propagation of the BiLSTM prediction model, Represents the output state obtained by the back propagation of the BiLSTM prediction model.

[0028] The propagation of forward LSTM or backward LSTM has the following relationship: ; in, represents the candidate state value, Indicates the state value at the previous moment, Indicates the current state value. Indicates the output value, Indicates the hidden state, Indicates the current input. represents the input gate, represents the forget gate, represents the output gate, represents the input weight matrix of the input gate, represents the hidden state weight matrix of the input gate, represents the bias term of the input gate, represents the input weight matrix of the forget gate, represents the hidden state weight matrix of the forget gate, Represents the bias term of the forget gate; represents the input weight matrix of the output gate, represents the hidden state weight matrix of the output gate, represents the bias term of the output gate; represents the input weight matrix of the candidate state, The hidden state weight matrix representing the candidate state, represents the bias term of the candidate state, and Represents two different activation functions.

[0029] Furthermore, the BiLSTM architecture consists of a forward LSTM and a backward LSTM, which together capture the temporal dependency of the input information. The forward LSTM processes the input to the viewport sequence in chronological order, while the backward LSTM processes the sequence in reverse chronological order; the outputs of the two will be concatenated and combined with the object detection information to form the final prediction result.

[0030] The BiLSTM network extends the unidirectional LSTM network with reverse time-order flow; it updates the current state by simultaneously forward propagating to obtain the output state and backward propagating to obtain the output state. Compared with the traditional unidirectional LSTM, BiLSTM can learn past and future information at the same time, thereby obtaining more robust feature information; After the model is initialized, the viewport trajectory information of the previous 5 seconds needs to be input into the prediction model to prevent the prediction model from inputting empty data when predicting the viewport for the first time. Before the viewport trajectory information is input into the model, random noise needs to be introduced into the historical coordinate data, and the data needs to be transformed and interpolated. The detailed steps are as follows: 1. Add random noise to coordinate: First, code inspection Are all elements of the array the same? If all elements are equal, it means The coordinates have not changed in the historical data. A small random noise is added to the coordinates to introduce a certain amount of variation to enhance the diversity and authenticity of the data; Then, go through each coordinates, check if there is an element that is zero; if a If the coordinate is zero, a larger random number is added to it to prevent the zero value from affecting subsequent mathematical operations; 2. Logarithmic transformation and difference calculation: After adding noise, Coordinates and The coordinates are logarithmically transformed and mapped to logarithmic space; this is to smooth the changes in the coordinate data and facilitate subsequent analysis and processing; After transformation Coordinates and The coordinates are calculated by first-order difference, so as to obtain the change of the coordinates between adjacent time steps; wherein the first-order difference is used to analyze the trend and characteristics of the coordinate change; This process introduces noise into the coordinate data and performs transformation and difference calculation, making the input data of the prediction model more random and stable, thereby enhancing the model's generalization ability and adaptability to different scenarios.

[0031] In step 2, the width of the viewport coordinates is adjusted according to the formed prediction result to obtain the final predicted future viewport coordinates; Width adjustment is when the viewport coordinates If the value exceeds the width of the video frame, it is adjusted to , to ensure that the coordinates are within the valid frame range; similarly, if If the value is negative, it is adjusted to ; Since it is difficult to cross the boundary in the vertical direction, only The coordinates are adjusted; In order to further filter the target objects related to the predicted viewport, the distance between the predicted viewport and the target object in each frame is sorted. From the sorted list, M is selected as the distance threshold. Any object below this distance threshold will be regarded as the target object of interest; in each frame, the selected target object is not fixed, but dynamically adjusted according to the distance. In this way, the position of the target object with the highest score in the original frame will have a certain impact on the viewport, which not only improves the accuracy of the prediction, but also enhances the user experience when watching 360° video.

[0032] In this embodiment, the edge server inputs the received user historical viewport trajectory information and target object trajectory information into the designed BiLSTM prediction model to obtain the predicted future viewport coordinates, and the user historical viewport trajectory information needs to be preprocessed before being input into the prediction model; The specific steps of preprocessing are as follows: Calculate the maximum number of video frames based on the timestamp of the viewport track; The window size for viewing 360° videos is set to 2560 in width and 1440 in height, and the viewport trajectory data is converted to the position coordinates on the corresponding window.

[0033] Initialize the BiLSTM prediction model. The detailed steps are as follows: First, the deep neural network model is initialized, using a sequence model architecture; the sequence model uses a bidirectional long short-term memory network layer, which has the ability to capture the time series characteristics of the input data; specifically, the first layer is a bidirectional LSTM layer, in which the number of hidden units is 128, and the layer is set Parameters to ensure that the layer can output a complete time series; the dimension of the input data is set to , that is, the input sequence length is 120, and the number of input features is 2; among them, is a parameter setting of the LSTM layer, which determines the output format of the layer. When , the LSTM layer will return the hidden state of each time step, and the output will contain all the hidden states of the entire time series. This setting is suitable for the scenario where BiLSTM needs to process the entire sequence bidirectionally; To prevent the model from overfitting, a regularization layer with a dropout rate of 0.1 is added immediately after the bidirectional LSTM layer. This regularization layer improves the generalization ability of the model by randomly ignoring the output of some neurons. The BiLSTM prediction model further adds a fully connected layer containing 64 neurons to perform nonlinear combination of the features extracted by the LSTM layer. In order to enhance the nonlinear representation capability of the fully connected layer, the activation function uses a rectified linear unit with leakage, where the initial parameter is set to 0.01, allowing some negative values ​​to pass through to retain gradient information. After the rectified linear unit activation function with leakage, another regularization layer is added with a dropout rate of 0.1 to further reduce the risk of overfitting; The final output layer of the BiLSTM prediction model is a fully connected layer with 60 neurons. tanh Activation function, the final output layer is responsible for generating the final output value, and the output dimension is 60; The BiLSTM prediction model is trained using an optimization algorithm for training deep learning models. The learning rate is set to 0.0001, which is an adaptive learning rate optimization algorithm that can effectively deal with the gradient vanishing problem. In addition, the loss function uses the mean absolute error to ensure that the model can regress and predict continuous values ​​more accurately.

[0034] Step 3. Based on the final predicted future viewport coordinates, the predicted information of the user's field of view is obtained. The predicted row and column coordinates of the tile group within the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group within the user's field of view to obtain the result of bit rate assignment. The result of bit rate assignment is requested by the edge server to the cloud server for video data caching and rendering in advance.

[0035] See also Figure 2 In step 3, based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the prediction information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The result of bit rate assignment is requested by the edge server to the cloud server for video data to be cached and rendered in advance, which specifically includes the following steps: Predicting the user's field of view through the final predicted future viewport coordinates to obtain predicted information of the user's field of view, and performing coordinate prediction on the predicted information of the user's field of view to obtain row and column coordinates of a tile group within the predicted user's field of view; Among them, the row and column coordinates of the tile group in the user's field of view are predicted, and the corresponding relationship in the process is as follows: ; in, Represents the horizontal coordinate of the predicted viewport, Represents the vertical coordinate of the predicted viewport, Indicates the number of columns of the video frame, Indicates the number of lines of the video frame, Indicates the column coordinates of the user's field of view, Indicates the row coordinates of the user's field of view, Indicates the video frame height; According to the allocation principle, the tiles within the user's field of view are assigned preset weights by predicting the row and column coordinates of the tile group, and the tile group outside the user's field of view is calculated to obtain the calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view area; The calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view is used to assign different weights to tiles at different distances outside the user's field of view through an allocation mechanism, and the bit rate is allocated according to the weights of tiles inside and outside the field of view; Among them, the calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view area is used to assign different weights to tiles at different distances outside the user's field of view through an allocation mechanism. The corresponding relationship in the process is as follows: ; in, Indicates Line The weight of the tile of the column, Represents the constructed weight matrix.

[0036] Furthermore, the bit rate decision in step 3 includes a function for calculating a relative position, the purpose of which is to determine the specific position of the viewport coordinates relative to the tiles in the video frame; by converting the viewport coordinates into tile coordinates on the video frame, the distribution of tiles in the viewport can be clarified, thereby affecting the bit rate allocation of these tiles in the adaptive stream; Based on the BiLSTM prediction results, the bit rate is allocated to each tile; the row and column coordinates of the tiles in the user's field of view are obtained; According to the designed allocation principle, the tiles within the field of view will be assigned a preset bit rate to ensure the picture quality of the corresponding area.

[0037] For tiles outside the field of view, the minimum distance between them and the boundary of the field of view is calculated. Based on this distance, a bit rate weight distribution mechanism is set, that is, tiles closer to the boundary of the field of view will get a higher bit rate weight, while tiles farther away will have a lower weight. This strategy aims to optimize the transmission efficiency of 360° videos by dynamically adjusting the bitrate of tiles in different regions. In particular, when dealing with the surround properties of 360° videos, special attention should be paid to the tiles at both ends of the video frame, which are physically farthest apart but adjacent in the user's visual experience. Therefore, when calculating the distance between tiles, a boundary determination mechanism is introduced. Moreover, all tiles in the video frame are arranged in 8x8, so an 8x8 weight matrix is ​​constructed, and the weight function is normalized so that the net weight of all frame blocks is 1, and then the bitrate is allocated to each tile according to the ratio specified in the algorithm. The edge server then requests video data of different bit rates from the cloud server and caches and renders them in advance. When the cloud server completes the target detection task, it transmits the detection results to the edge server, where viewport prediction, rendering and encoding are performed, thereby reducing transmission delay and bandwidth usage. Furthermore, when calculating the distance between tiles, a boundary determination mechanism is introduced. The corresponding relationship of the process is as follows: ; in, express The horizontal coordinate of the viewpoint position at the moment, express The horizontal coordinate of the viewpoint position at the moment, Indicates the width of the video frame. Indicates the situation of judgment; It should be noted that when the first type of judgment condition is met, it means that the viewport coordinates move from the right boundary of the video to the left boundary; when the second type of judgment condition is met, it means that the viewport coordinates move from the left boundary of the video to the right boundary; when the third type of judgment condition is met, it means that the viewport coordinates move normally within the video and no cross-border situation occurs.

[0038] Step 4: The client sends a request to the edge server to obtain predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

[0039] In step 4, the method of the present invention further includes: calculating a user experience quality index according to the quality of the video watched by the user; Calculating the user experience quality index based on the quality of the video watched by the user specifically includes the following steps: Calculate the average absolute error between the predicted viewport coordinates and the actual viewport coordinates. The corresponding relationship is as follows: ; in, represents the mean absolute error, Represents the number of viewport coordinates output by the prediction model, Indicates The horizontal coordinate of the predicted viewport, Indicates The vertical coordinate of the predicted viewport, Indicates The horizontal coordinate of the actual viewport, Indicates The ordinate of the actual viewport; The deviation between the tile coordinates calculated from the predicted viewport coordinates and the tile coordinates calculated from the actual viewport coordinates is measured. The relationship between the corresponding process is as follows: ; in, Indicates a tile error. Indicates the total number of columns in the video frame. Indicates the total number of rows. Indicates the video frame width, Indicates the video frame height; The ratio between the predicted tiles within the user's field of view and the tiles actually viewed by the user is measured. The corresponding relationship is as follows: ; in, Indicates tile coverage, Represents the predicted set of tiles within the user's field of view, Represents the set of tiles that the user actually views; To measure the model performance, we define QoE The quality function is used to empirically evaluate the model performance. QoE The quality function is defined as the weighted sum of four playback functions: ; in, Indicates The average bit rate of the tiles covered by the user's viewport in the video segment, Indicates the quality change within the frame, Indicates the change in quality of consecutive frames. Indicates the quality change between video segments; In 360° video processing, the video is evenly divided into Analyze each segment. For the The average bit rate of tiles covered by the user's viewport in a video segment; at the same time, pay attention to the quality changes within the frame , that is, the fluctuation of the average bit rate of the viewport tiles in the same video segment, to avoid excessive quality fluctuations affecting the user's viewing experience; in addition, the quality changes of consecutive frames Pay attention to the quality stability between adjacent frames to prevent the picture quality from fluctuating and causing users to feel dizzy; finally, the quality changes between video segments We focus on the smooth transition of viewport bitrate between video segments to reduce quality fluctuations and further enhance the user's immersive experience.

[0040] It should be noted that the mean absolute error between the predicted viewport coordinates and the actual viewport coordinates can intuitively indicate the size of the prediction error of the model.

[0041] See also Figure 5 The present invention also proposes a 360° video edge-assisted viewport prediction and transmission system based on BiLSTM, the system comprising: Video preprocessing module for: The 360° video stored in the cloud server is divided into several segments, and each frame of each video segment is divided and projected to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the target object motion trajectory information is generated according to the object coordinate sequence, and the target object motion trajectory information is sent to the edge server; Edge-viewport prediction module, used to: The client sends the user's historical viewport information to the edge server, and the edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result, and the prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates; Edge-precaching decision module, used to: Based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The edge server requests the cloud server to cache and render the video data in advance based on the result of bit rate assignment. Client-video playback module, used for: The client sends a request to the edge server to obtain the predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

[0042] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0043] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0044] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A 360° video edge-assisted viewport prediction method based on BiLSTM, characterized in that: The method comprises the following steps: Step 1: Divide the 360° video stored in the cloud server into several segments, and project each frame of each segment to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the motion trajectory information of the target object is generated according to the coordinate sequence of the object, and the motion trajectory information of the target object is sent to the edge server; Step 2: The client sends the user's historical viewport information to the edge server. The edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result. The prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates. Step 3: Based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The edge server requests the cloud server for video data in advance for caching and rendering according to the result of bit rate assignment. Step 4: The client sends a request to the edge server to obtain predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

2. According to claim 1, a 360° video edge-assisted viewport prediction method based on BiLSTM is characterized in that: In step 1, the 360° video stored in the cloud server is divided into several segments, and each frame of each segment of the video is divided and projected to obtain an expanded image, the expanded image is detected by a detection algorithm to obtain a coordinate sequence of the object, and the motion trajectory information of the target object is generated according to the coordinate sequence of the object, which specifically includes the following steps: The 360° video is segmented in the time dimension to obtain segmented video frames, and the segmented video frames are mapped on a two-dimensional plane to obtain an unfolded image; The YOLOv3 target detection algorithm is used to detect the unfolded image to obtain the detection results, the detection results are processed by boundaries and assigned numbers, and the coordinate information of the objects to which the number assignment is completed is recorded. The coordinate sequence of the objects with corresponding numbers is generated according to the recorded coordinate information; Among them, the boundary processing process corresponds to the following relationship: ； in, Indicates The horizontal coordinate of the target object, Indicates the video frame width; The coordinate sequences of the objects with corresponding numbers are connected to track the motion trajectory of the numbered objects to obtain the trajectory result, and the tracked trajectory result is sent to the edge server.

3. According to claim 2, a 360° video edge-assisted viewport prediction method based on BiLSTM is characterized in that: In step 2, the client sends the user's historical viewport information to the edge server, and the edge server preprocesses the user's historical viewport information and the target object motion trajectory information, and inputs the preprocessing result into the BiLSTM prediction model to obtain the prediction result, and combines the prediction result with the user's historical viewport information and the target object motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates, which specifically includes the following steps: The client sends the user's historical viewport trajectory information to the edge server, and the edge server pre-processes the received user's historical viewport trajectory information; Initialize the BiLSTM prediction model and send the user's historical viewport trajectory information preprocessed by the edge server to the initialized BiLSTM prediction model; The forward LSTM of the BiLSTM prediction model is processed in chronological order and input into the viewport sequence, and the backward LSTM of the BiLSTM prediction model is processed in the reverse order of chronological order and input into the viewport sequence. The forward LSTM and the backward LSTM jointly process the received information and detect the expanded image to obtain the detection result to form the prediction result.

4. According to claim 3, a 360° video edge-assisted viewport prediction method based on BiLSTM is characterized in that: The client sends the user's historical viewport trajectory information to the edge server, and the edge server preprocesses the received user's historical viewport trajectory information, specifically including the following steps: The maximum number of video frames is calculated based on the timestamp of the target object's motion trajectory. The corresponding process has the following relationship: ； in, Represents iteration Find the value with the largest timestamp. Indicates the number of frames per second. Indicates the timestamp, Indicates the maximum number of frames per second. Indicates converting the calculation result into an integer; The viewport trajectory data is converted to the position coordinates on the corresponding window. The corresponding process has the following relationship: ； in, express The values ​​of the axis coordinates, express The values ​​of the axis coordinates, Represents the original viewport of the current frame Axis coordinate scale information, Represents the original viewport of the current frame Axis coordinate scale information, Indicates the video frame width, Indicates the video frame height, Indicates the original width of the 360° video. Indicates the original height of the 360° video.

5. According to claim 4, a 360° video edge-assisted viewport prediction method based on BiLSTM is characterized in that: In the step of processing the forward LSTM of the BiLSTM prediction model in chronological order and inputting it into the viewport sequence, processing the backward LSTM of the BiLSTM prediction model in the reverse order of chronological order and inputting it into the viewport sequence, and processing the received information by the forward LSTM and the backward LSTM together to detect the expanded image to obtain the detection result to form the prediction result, the corresponding relationship between the process is as follows: ； in, Indicates the current status. represents the forward output state, Indicates the backward output state, Indicates the current time step The feature vector input is represents the forward LSTM propagation along the time direction, represents the backward LSTM propagation along time, Represents the output state obtained by the forward propagation of the BiLSTM prediction model, Represents the output state obtained by the back propagation of the BiLSTM prediction model.

6. The 360° video edge-assisted viewport prediction method based on BiLSTM according to claim 5, characterized in that: The propagation of forward LSTM or backward LSTM has the following relationship: ； in, represents the candidate state value, Indicates the state value at the previous moment, Indicates the current state value. Represents the output value, Indicates the hidden state, Indicates the current input. represents the input gate, represents the forget gate, represents the output gate, represents the input weight matrix of the input gate, represents the hidden state weight matrix of the input gate, represents the bias term of the input gate, represents the input weight matrix of the forget gate, represents the hidden state weight matrix of the forget gate, Represents the bias term of the forget gate; represents the input weight matrix of the output gate, represents the hidden state weight matrix of the output gate, Represents the bias term of the output gate; represents the input weight matrix of the candidate state, The hidden state weight matrix representing the candidate state, represents the bias term of the candidate state, and Represents two different activation functions.

7. The 360° video edge-assisted viewport prediction method based on BiLSTM according to claim 6, characterized in that: In the step 3, based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the prediction information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of the bit rate assignment. The result of the bit rate assignment is requested by the edge server to the cloud server for video data to be cached and rendered in advance, which specifically includes the following steps: Predicting the user's field of view through the final predicted future viewport coordinates to obtain predicted information of the user's field of view, and performing coordinate prediction on the predicted information of the user's field of view to obtain row and column coordinates of a tile group within the predicted user's field of view; Among them, the row and column coordinates of the tile group in the user's field of view are predicted, and the corresponding relationship in the process is as follows: ； in, Represents the horizontal coordinate of the predicted viewport, Represents the vertical coordinate of the predicted viewport, Indicates the number of columns of the video frame, Indicates the number of lines in the video frame, Indicates the column coordinates of the user's field of view, Indicates the row coordinates of the user's field of view, Indicates the video frame height; According to the allocation principle, the tiles within the user's field of view are assigned preset weights by predicting the row and column coordinates of the tile group, and the tile group outside the user's field of view is calculated to obtain the calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view area; The calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view is used to assign different weights to tiles at different distances outside the user's field of view through an allocation mechanism, and the bit rate is allocated according to the weights of tiles inside and outside the field of view; Among them, the calculation result of the minimum distance between the tile group outside the user's field of view and the boundary of the field of view area is used to assign different weights to tiles at different distances outside the user's field of view through an allocation mechanism. The corresponding relationship in the process is as follows: ； in, Indicates Line The weight of the tile of the column, Represents the constructed weight matrix.

8. The 360° video edge-assisted viewport prediction method based on BiLSTM according to claim 7, characterized in that: In step 4, the method further includes: calculating a user experience quality index according to the quality of the video watched by the user; Calculating the user experience quality index based on the quality of the video watched by the user specifically includes the following steps: Calculate the mean absolute error between the predicted viewport coordinates and the actual viewport coordinates. The corresponding relationship is as follows: ； in, represents the mean absolute error, Represents the number of viewport coordinates output by the prediction model, Indicates The horizontal coordinate of the predicted viewport, Indicates The vertical coordinate of the predicted viewport, Indicates The horizontal coordinate of the actual viewport, Indicates The ordinate of the actual viewport; The deviation between the tile coordinates calculated from the predicted viewport coordinates and the tile coordinates calculated from the actual viewport coordinates is measured. The relationship between the corresponding process is as follows: ； in, Indicates a tile error. Indicates the total number of columns in the video frame. Indicates the total number of rows, Indicates the video frame width, Indicates the video frame height; The ratio between the predicted tiles within the user's field of view and the tiles actually viewed by the user is measured. The corresponding relationship is as follows: ； in, Indicates tile coverage, Represents the predicted set of tiles within the user's field of view, Represents the set of tiles that the user actually views; Measuring model performance,We empirically evaluate model performance using a defined QoE quality function, which for each video segment is defined as the weighted sum of four playback functions: ； in, Indicates The average bit rate of the tiles covered by the user's viewport in the video segment, Indicates the quality change within the frame, Indicates the change in quality of consecutive frames. Indicates the quality variation between video segments.

9. A 360° video edge-assisted viewport prediction system based on BiLSTM, characterized in that: The system applies any one of the BiLSTM-based 360° video edge-assisted viewport prediction methods according to claims 1 to 8, and the system comprises: Video preprocessing module for: The 360° video stored in the cloud server is divided into several segments, and each frame of each video segment is divided and projected to obtain an expanded image. The expanded image is detected by a detection algorithm to obtain the coordinate sequence of the object, and the target object motion trajectory information is generated according to the object coordinate sequence, and the target object motion trajectory information is sent to the edge server; Edge-viewport prediction module, used to: The client sends the user's historical viewport information to the edge server, and the edge server preprocesses the user's historical viewport information and the target object's motion trajectory information. The preprocessing result is input into the BiLSTM prediction model to obtain the prediction result, and the prediction result is combined with the user's historical viewport information and the target object's motion trajectory information to obtain the final prediction result. The final prediction result is obtained by adjusting the viewport coordinate value in the BiLSTM prediction model to obtain the final predicted future viewport coordinates; Edge-precaching decision module, used to: Based on the final predicted future viewport coordinates, the prediction information of the user's field of view is obtained, and the predicted row and column coordinates of the tile group in the user's field of view are obtained according to the predicted information of the user's field of view. The bit rate is assigned to the video division parts inside and outside the field of view through the predicted row and column coordinates of the tile group in the user's field of view to obtain the result of bit rate assignment. The edge server requests the cloud server to cache and render the video data in advance based on the result of bit rate assignment. Client-video playback module, used for: The client sends a request to the edge server to obtain the predicted video information based on the video information that starts playing, and obtains the next corresponding predicted viewport video content.

Images