Panoramic intelligent playing system based on multi-dimensional data fusion

Abstract

The invention discloses a panoramic intelligent playing system based on multi-dimensional data fusion, and relates to the field of video processing. Comprising a data acquisition module for acquiring panoramic video data including a video stream, an audio stream, sensor data and user interaction data; the data fusion module is used for fusing the video stream, the audio stream, the sensor data and the user interaction data by adopting a multi-dimensional data fusion algorithm; the intelligent playing module intelligently adjusts playing parameters to perform panoramic video playing according to the fused data F; and the interaction feedback module feeds back a playing state in real time according to user interaction data U. The sensor data is a gyroscope or an accelerometer, and the user interaction data comprises clicking, sliding, voice instructions and gesture instructions. According to the method, the playing quality is improved, smooth playing and intelligent switching of the panoramic video are realized through fusion processing of the multi-dimensional data, and the playing quality is improved.

Claims

1. A panoramic intelligent playback system based on multi-dimensional data fusion, characterized in that: include: Data acquisition module, which collects panoramic video data, including video stream, audio stream, sensor data and user interaction data; The data fusion module uses a multi-dimensional data fusion algorithm to fuse video streams, audio streams, sensor data, and user interaction data; The intelligent playback module intelligently adjusts playback parameters to play panoramic videos based on the fused data F; The interactive feedback module provides real-time feedback on the playback status based on the user interaction data U.

2. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 1, characterized in that: The sensor data is a gyroscope or an accelerometer.

3. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 1, characterized in that: The user interaction data includes clicks, slides, voice commands, and gesture commands.

4. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 1, characterized in that: The data fusion module, which fuses data, includes the following steps: A1: Data preprocessing, which involves data cleaning, data standardization, and data dimension reduction of the collected data; A2: Extract features, perform feature extraction on multi-dimensional data, and extract the features of each dimension of data; A3: Select a fusion strategy and perform multi-dimensional fusion of the extracted features.

5. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 4, characterized in that: In step A1, data dimension reduction is performed using principal component analysis or linear discriminant analysis.

6. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 5, characterized in that: The principal component analysis comprises the following steps: A11a: Data standardization: First, the data is standardized so that the mean of each feature is 0 and the variance is 1. A12a: Calculate the covariance matrix. Calculate the covariance matrix of the sample, whose elements represent the covariance between each feature; A13a: Calculate eigenvalues ​​and eigenvectors. Extract eigenvalues ​​and eigenvectors from the covariance matrix. Eigenvalues ​​indicate the importance of the corresponding eigenvectors, while eigenvectors define the direction of dimensionality reduction of the data. A14a: Sort the eigenvectors by their eigenvalues ​​from large to small, and select the eigenvectors corresponding to the first k largest eigenvalues ​​as the principal components; A15a: Project the original data into a new space composed of the selected principal components to achieve data dimensionality reduction.

7. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 5, characterized in that: The linear discriminant analysis comprises the following steps: A11b: Data classification, classify the data; A12b: Category mean calculation, calculate the mean vector in each category of data; A13b: Overall mean calculation, calculate the overall mean vector of all samples; A14b: Calculation of inter-class and intra-class scatter matrices. The intra-class scatter matrix is ​​used to measure the degree of dispersion of samples within the same class, while the inter-class scatter matrix is ​​used to measure the degree of dispersion of samples between different classes. A15b: Solve the generalized eigenvalue and eigenvector. By solving the generalized eigenvalue problem, find the eigenvector corresponding to the maximum generalized eigenvalue. The eigenvector defines the dimensionality reduction direction of the data, that is, the LDA feature space. A16b: Select the projection direction, and select the eigenvectors corresponding to the first k largest generalized eigenvalues ​​as the projection direction; A17b: Project the original data into a new space formed by the selected projection directions to achieve data dimensionality reduction.

8. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 4, characterized in that: In step A2, the extraction of video features includes the following steps: A21a: Keyframe extraction, using color histogram or edge detection to identify change points in the video and determine the keyframes; A22a: Image feature extraction, using convolutional neural network (CNN) to extract color, texture and shape features of images; A23a: Video sequence feature extraction, using any of the optical flow method, background subtraction, and motion estimation methods to extract motion features in the video as well as the tempo features of the video and shot switching.

9. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 4, characterized in that: In step A2, the feature extraction of audio data includes the following steps: A21b: Preprocessing: denoising, framing, and windowing audio data; A22b: Time domain feature extraction, extracting the short-time energy, short-time average zero-crossing rate, and autocorrelation function of the audio signal in the time domain; A23b: Frequency domain feature extraction: Perform Fourier transform or fast Fourier transform on the audio signal to convert the time domain signal into a frequency domain signal, and extract the spectrum, spectrum centroid, and spectrum roll-off point features from the frequency domain signal; A24b: Cepstrum feature extraction, using filter banks or discrete cosine transform to extract sister frequency cepstrum coefficients; A25b: Advanced feature extraction, extracting phoneme features from speech signals and emotional features for emotion recognition.

10. The panoramic intelligent playback system based on multi-dimensional data fusion according to claim 4, characterized in that: In step A3, the fusion strategy is a Bayesian fusion function or a deep learning fusion function; The model expression of the Bayesian fusion function is P(H i | Data) is the posterior probability, representing the probability of hypothesis H i under the observed data Data; The expression of the deep fusion model is y=f(x1, x2, ..., x n ), y is the prediction result or decision result, x1, x2, ..., x n are features extracted from different data sources.

Images