Panoramic video processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a panoramic video processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a panoramic video frame sequence and a panoramic audio clip sequence of a panoramic video to be processed; performing feature extraction operation on the panoramic video frame in sequence, wherein the feature extraction operation comprises the steps of performing visual feature extraction on the current panoramic video frame, fusing a video feature extraction result with a historical visual state to obtain a space-time visual feature of the current panoramic video frame, and updating the historical visual state based on the space-time visual feature; performing auditory feature extraction on each panoramic audio clip to obtain corresponding auditory features; fusing the space-time visual features of the panoramic video frames with the corresponding auditory features to obtain corresponding fused audio-visual features; and performing saliency prediction based on the fused audio-visual features of each panoramic video frame to obtain a corresponding saliency distribution diagram. The computing resource consumption and the time overhead of significance prediction are reduced, and the prediction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a panoramic video processing method, device, electronic equipment and storage medium. Background Art

[0002] Omnidirectional video (ODV) is a video that provides a 360-degree omnidirectional view. It usually has a huge amount of data, which poses challenges to video storage, transmission, and rendering. Therefore, it is necessary to find the area of ​​interest (i.e., the area of ​​visual attention) of the human eye in the panoramic video to reduce the redundant information in the video. Saliency prediction refers to determining the area of ​​interest in an image or video. These areas of interest usually contain important objects (such as people or objects) or are related to high-level semantics.

[0003] The saliency prediction method for panoramic videos in the related art not only consumes a large amount of computing resources and is time-consuming, but also has poor accuracy in saliency prediction in complex sound scenes. Summary of the invention

[0004] In order to solve the problems of the prior art, the embodiments of the present invention provide a panoramic video processing method, device, electronic device and storage medium. The technical solution is as follows:

[0005] On the one hand, a panoramic video processing method is provided, the method comprising:

[0006] Obtain a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed;

[0007] Performing a feature extraction operation on the panoramic video frames in the panoramic video frame sequence in sequence, the feature extraction operation comprising: extracting visual features on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame;

[0008] Extracting auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment;

[0009] Fusing the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audiovisual features corresponding to each panoramic video frame;

[0010] Based on the fused audio-visual features corresponding to each of the panoramic video frames, saliency prediction is performed respectively to obtain a saliency distribution map corresponding to each of the panoramic video frames.

[0011] In another aspect, a panoramic video processing device is provided, the device comprising:

[0012] A frame sequence acquisition module, used to acquire a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed;

[0013] a spatiotemporal visual feature extraction module, configured to sequentially perform feature extraction operations on the panoramic video frames in the panoramic video frame sequence, wherein the feature extraction operations include: extracting visual features on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame;

[0014] an auditory feature extraction module, configured to extract auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment;

[0015] An audio-visual fusion module, used to fuse the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audio-visual features corresponding to each panoramic video frame;

[0016] The saliency prediction module is used to perform saliency prediction based on the fused audio-visual features corresponding to each panoramic video frame to obtain a saliency distribution map corresponding to each panoramic video frame.

[0017] In an exemplary embodiment, the audio-visual fusion module includes:

[0018] a channel stitching module, configured to perform channel stitching on the spatiotemporal visual features corresponding to each panoramic video frame and the auditory features of the panoramic audio clip corresponding to the panoramic video frame, so as to obtain the channel stitching features corresponding to the panoramic video frame;

[0019] The first convolution processing module is used to perform convolution processing on the channel stitching features corresponding to the panoramic video frame to obtain the fused audio-visual features corresponding to the panoramic video frame.

[0020] In an exemplary embodiment, the auditory feature extraction module includes:

[0021] A mel feature generation module, configured to generate a mel feature corresponding to each panoramic audio segment in the panoramic audio segment sequence;

[0022] The auditory feature extraction submodule is used to extract auditory features from the Mel features corresponding to the panoramic audio segment to obtain the auditory features of the panoramic audio segment.

[0023] In an exemplary embodiment, the frame sequence acquisition module includes:

[0024] A video frame sequence determination module is used to obtain a panoramic video to be processed, read each video frame in the panoramic video to be processed in sequence, and obtain a panoramic video frame sequence corresponding to the panoramic video to be processed;

[0025] an initial audio segment determination module, configured to obtain, for each panoramic video frame in the panoramic video frame sequence, audio information corresponding to a time period between a first playback time point and a second playback time point from the panoramic video to be processed based on a playback time point of the panoramic video frame in the panoramic video to be processed, to obtain an initial panoramic audio segment corresponding to the panoramic video frame; the first playback time point is located before the playback time point and is separated from the playback time point by a first preset time length, and the second playback time point is located after the playback time point and is separated from the playback time point by a second preset time length;

[0026] The audio segment determination module is used to convert the audio information of the initial panoramic audio segment corresponding to the panoramic video frame into mono, so as to obtain the panoramic audio segment corresponding to the panoramic video frame.

[0027] In an exemplary embodiment, the apparatus further comprises:

[0028] A sample acquisition module, used to acquire a sample panoramic video frame sequence and a sample panoramic audio clip sequence corresponding to a sample panoramic video, each sample panoramic video frame corresponding to a reference saliency distribution map;

[0029] A sample spatiotemporal visual feature extraction module is used to sequentially input the sample panoramic video frames in the sample panoramic video frame sequence into the spatiotemporal visual feature extraction network of the initial panoramic video processing network to perform a sample feature extraction operation, wherein the sample feature extraction operation includes: performing sample visual feature extraction on the current sample panoramic video frame, fusing the sample visual feature extraction result with the historical sample visual state to obtain the sample spatiotemporal visual feature of the current sample panoramic video frame, and updating the historical sample visual state based on the sample spatiotemporal visual feature of the current sample panoramic video frame;

[0030] A sample auditory feature extraction module, configured to extract auditory features from each sample panoramic audio segment in the sample panoramic audio segment sequence based on the auditory feature extraction network of the initial panoramic video processing network, so as to obtain sample auditory features of each sample panoramic audio segment;

[0031] A sample audio-visual fusion module is used to fuse the sample spatiotemporal visual features of each of the sample panoramic video frames with the sample auditory features of the corresponding sample panoramic audio clips to obtain sample fused audio-visual features corresponding to each of the sample panoramic video frames;

[0032] A prediction module, configured to perform saliency prediction on the sample fused audiovisual features corresponding to each of the sample panoramic video frames based on the saliency prediction network of the initial panoramic video processing network, and obtain a predicted saliency distribution map corresponding to each of the sample panoramic video frames;

[0033] A training module is used to adjust the network parameters of the initial panoramic video processing network based on the difference between the predicted saliency distribution map corresponding to each of the sample panoramic video frames and the corresponding reference saliency distribution map until a preset training end condition is met to obtain the panoramic video processing network.

[0034] In an exemplary embodiment, the sample spatiotemporal visual feature extraction module includes:

[0035] a second convolution processing module, configured to perform convolution processing on the current sample panoramic video frame based on the first convolution processing network in the spatiotemporal visual feature extraction network, and use the result of the convolution processing as the sample visual feature extraction result;

[0036] A first weight system determination module, used to obtain the current network parameters of the time domain network in the spatiotemporal visual feature extraction network, and determine the weight coefficient of the sample visual feature extraction result based on the current network parameters of the time domain network;

[0037] The first weighted processing module is used to perform weighted summation processing on the sample visual feature extraction result and the historical sample visual state based on the weight coefficient of the sample visual feature extraction result to obtain the sample spatiotemporal visual features of the current sample panoramic video frame; wherein the sum of the weight coefficient of the sample visual feature extraction result and the weight coefficient of the historical sample visual state is a preset value.

[0038] In an exemplary embodiment, the sample audio-visual fusion module includes:

[0039] A sample channel stitching module is used to perform channel stitching on the sample spatiotemporal visual features corresponding to each of the sample panoramic video frames and the sample auditory features of the sample panoramic audio clips corresponding to the sample panoramic video frames, so as to obtain the sample channel stitching features corresponding to the sample panoramic video frames;

[0040] The third convolution processing module is used to input the sample channel stitching features corresponding to the sample panoramic video frame into the second convolution processing network of the initial panoramic video processing network for convolution processing to obtain the sample fusion audio-visual features corresponding to the sample panoramic video frame.

[0041] In an exemplary embodiment, the sample auditory feature extraction module includes:

[0042] A sample Mel feature generation module, configured to generate, for each sample panoramic audio segment in the sample panoramic audio segment sequence, a sample Mel feature corresponding to the sample panoramic audio segment;

[0043] The sample auditory feature extraction submodule is used to input the sample Mel features corresponding to the sample panoramic audio clip into the auditory feature extraction network of the initial panoramic video processing network to extract auditory features, so as to obtain the sample auditory features of the sample panoramic audio clip.

[0044] In an exemplary embodiment, the sample acquisition module includes:

[0045] A sample video frame determination module is used to obtain a sample panoramic video, read each video frame in the sample panoramic video in sequence, and obtain a sample panoramic video frame sequence corresponding to the sample panoramic video;

[0046] An initial sample audio segment determination module is used to obtain, for each sample panoramic video frame in the sample panoramic video frame sequence, audio information corresponding to a time period between a first sample playback time point and a second sample playback time point from the sample panoramic video based on a sample playback time point of the sample panoramic video frame in the sample panoramic video, so as to obtain an initial sample panoramic audio segment corresponding to the sample panoramic video frame; the first sample playback time point is located before the sample playback time point and is separated from the sample playback time point by a first preset time length, and the second sample playback time point is located after the sample playback time point and is separated from the sample playback time point by a second preset time length;

[0047] The sample panoramic audio segment determination module is used to convert the audio information of the initial sample panoramic audio segment corresponding to the sample panoramic video frame into mono, so as to obtain the sample panoramic audio segment corresponding to the sample panoramic video frame.

[0048] In an exemplary embodiment, when the spatiotemporal visual feature extraction module extracts visual features of the current panoramic video frame and fuses the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, it is specifically used to: perform convolution processing on the current panoramic video frame based on the first convolution processing network in the panoramic video processing network, and use the result of the convolution processing as the visual feature extraction result; obtain the network parameters of the time domain network in the panoramic video processing network, and determine the weight coefficient of the visual feature extraction result based on the network parameters; based on the weight coefficient of the visual feature extraction result, perform weighted summation processing on the visual feature extraction result and the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame; wherein the sum of the weight coefficient of the visual feature extraction result and the weight coefficient of the historical visual state is the preset value.

[0049] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the panoramic video processing method of any of the above aspects.

[0050] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement a panoramic video processing method as described in any of the above aspects.

[0051] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned panoramic video processing methods.

[0052] The embodiment of the present application sequentially performs feature extraction operations on panoramic video frames in a panoramic video frame sequence, and the feature extraction operations include: performing visual feature extraction on the current panoramic video frame, fusing the visual feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame, and performing auditory feature extraction on each panoramic audio segment in a panoramic audio segment sequence to obtain the auditory features of each panoramic audio segment, and then fusing the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio segment to obtain the fused audio-visual features corresponding to each panoramic video frame, and performing saliency prediction based on the fused audio-visual features of each panoramic video frame to obtain the saliency distribution map corresponding to each panoramic video frame, thereby effectively predicting the visual attention in the panoramic video, which not only reduces the computing resource consumption and time overhead of saliency prediction, but also improves the accuracy of saliency prediction in complex sound scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0055] Figure 2 It is a flowchart of a panoramic video processing method provided by an embodiment of the present application;

[0056] Figure 3 is a flowchart of another panoramic video processing method provided in an embodiment of the present application;

[0057] Figure 4 is a flowchart of another panoramic video processing method provided in an embodiment of the present application;

[0058] Figure 5 is a flowchart of another panoramic video processing method provided in an embodiment of the present application;

[0059] Figure 6 is a schematic diagram of the structure of an initial panoramic video processing network provided in an embodiment of the present application;

[0060] Figure 7 is an example of a sample panoramic video frame and its corresponding reference saliency distribution map provided in an embodiment of the present application;

[0061] Figure 8 is a schematic diagram of the structure of another initial panoramic video processing network provided in an embodiment of the present application;

[0062] Fig. 9 is a schematic diagram of the structure of another initial panoramic video processing network provided in an embodiment of the present application;

[0063] Fig.10 This is a flow example of another panoramic video processing method provided in an embodiment of the present application;

[0064] Fig.11 is a structural block diagram of a panoramic video processing device provided in an embodiment of the present application;

[0065] Fig.12 It is a hardware structure block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0068] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0069] See also Figure 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present application, the implementation environment includes a terminal 110 and a server 120, wherein the terminal 110 and the server 120 can communicate via a wired or wireless network connection.

[0070] The terminal 110 includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, etc. The terminal 110 is installed with client software such as an application (Application, referred to as App) having a video processing function, which can be an independent application or a subroutine in the application. In the embodiment of the present invention, the processing object of the video processing function can include a panoramic video.

[0071] The server 120 can provide background services for the application in the terminal 110, and the background services can specifically be video processing services. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] In an exemplary embodiment, the terminal 110 and the server 120 can both be node devices in the blockchain system, and can share the acquired and generated information with other node devices in the blockchain system to achieve information sharing between multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which is composed of multiple blocks, and the adjacent blocks have an association relationship, so that when the data in any block is tampered, it can be detected by the next block, thereby preventing the data in the blockchain from being tampered with, and ensuring the security and reliability of the data in the blockchain.

[0073] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0074] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0075] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0076] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0077] See also Figure 2 , which is a flow chart of a panoramic video processing method provided by an embodiment of the present invention, which can be applied to Figure 1The electronic device shown. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or product is actually executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:

[0078] S201, obtaining a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed.

[0079] The panoramic video frames in the panoramic video frame sequence are arranged according to the order in which they are played in the panoramic video to be processed.

[0080] The panoramic audio segment sequence is obtained based on the audio information of the panoramic video to be processed, and the panoramic audio segment sequence corresponds to the panoramic video frame sequence one by one, that is, each panoramic video frame corresponds to a panoramic audio segment. Specifically, each panoramic audio segment may include audio information of a preset duration before and after the corresponding panoramic video frame. For example, the preset duration may be 0.5s, that is, each panoramic audio segment may include audio information of 0.5s before the corresponding panoramic video frame and audio information of 0.5s after the panoramic video frame, so that each panoramic audio segment contains 1s of audio information.

[0081] It should be noted that, for the panoramic audio segments corresponding to the beginning and the end of the panoramic video to be processed, a time domain zero-padding operation may be performed on the corresponding audio, so that each panoramic audio segment contains audio information of the same duration.

[0082] In the embodiment of the present application, the panoramic video frame may be a video frame in ERP (Equirectangular Projection) format separated from the panoramic video to be processed. The ERP format is a mainstream panoramic video projection format. Directly using the ERP format can save the huge amount of calculation and time overhead caused by the projection transformation. In addition, the position information of the pixel points is of great significance in the subsequent saliency prediction. The ERP format better preserves this information through the latitude and longitude network, which is conducive to improving the efficiency and accuracy of panoramic video processing.

[0083] S203, performing a feature extraction operation on the panoramic video frames in the panoramic video frame sequence in sequence, the feature extraction operation including: performing visual feature extraction on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame.

[0084] Specifically, assuming that the current panoramic video frame is panoramic video frame n-1, that is, the n-1th panoramic video frame in the panoramic video frame sequence, then the visual feature extraction is first performed on the panoramic video frame n-1, and then the visual feature extraction result is fused with the historical visual state to obtain the spatiotemporal visual features of the panoramic video frame n-1, and then the historical visual state is updated based on the spatiotemporal visual features of the panoramic video frame n-1, so that the historical visual state is the spatiotemporal visual features of the panoramic video frame n-1. Next, the feature extraction operation is performed on the panoramic video frame n, and the historical visual state fused with the visual feature extraction result in the feature extraction operation is actually the spatiotemporal visual features of the above-mentioned panoramic video frame n-1, so that on the basis of keeping the number of feature channels unchanged, the time domain information is integrated into each visual feature extraction result in an exponential translation fusion manner to obtain the corresponding spatiotemporal visual features.

[0085] Specifically, when visual feature extraction is performed on the current panoramic video frame, a convolution calculation may be performed on the current panoramic video frame, and the convolution calculation result is used as the visual feature extraction result.

[0086] Exemplarily, when the video feature extraction result is fused with the historical visual state to obtain the spatiotemporal visual feature of the current panoramic video frame, the weight coefficient corresponding to the video feature extraction result can be obtained, and then based on the weight coefficient corresponding to the video feature extraction result, the video feature extraction result and the historical visual state are weighted and summed, and the weighted summation result is used as the spatiotemporal visual feature. Among them, the sum of the weight coefficient of the video feature extraction result and the weight coefficient of the historical visual state is a preset value (such as a value of 1). In practical applications, the spatiotemporal visual feature can be obtained using the following formula (1):

[0087] f v (n) = α·C(n) + (1-α)·f v (n-1) (1)

[0088] Among them, f v (n) is the spatiotemporal visual feature of the current panoramic video frame; C(n) is the visual feature extraction result of the current panoramic video frame; f v (n-1) represents the historical visual state; α is the weight coefficient of the video feature extraction result corresponding to the current panoramic video frame; (1-α) is the weight coefficient corresponding to the historical visual state.

[0089] The weight coefficient α in the above formula (1) may be a preset value. Since usually, there should be a positive correlation between the previous and next frames of a video, the value range of the weight coefficient α is between 0 and 1.

[0090] In another possible implementation, the weight coefficient α in formula (1) can also be learned as a network parameter, so as to improve the fusion effect in the feature extraction operation. This implementation will be described in detail later in conjunction with the training of the panoramic video processing network.

[0091] In an embodiment of the present application, before visual features are extracted from the current panoramic video frame, the current panoramic video frame may be preprocessed to obtain an input image with a preset aspect ratio. For example, the current panoramic video frame in ERP format may be preprocessed into an input image with an aspect ratio of 1:2.

[0092] S205: extracting auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment.

[0093] In the embodiment of the present application, in order to improve the accuracy of auditory feature extraction, Figure 3 As shown, the above step S205 may include:

[0094] S301: For each panoramic audio segment in a panoramic audio segment sequence, generate a Mel feature corresponding to the panoramic audio segment.

[0095] S303: Perform auditory feature extraction on the Mel features corresponding to the panoramic audio clip to obtain auditory features of the panoramic audio clip.

[0096] The Mel feature may be a Mel spectrogram or a Mel cepstral coefficient (Mel-scale Frequency Cepstral Coefficients, MFCC).

[0097] The horizontal axis of the Mel spectrogram is the time scale, the vertical axis is the frequency scale, and the pixel value represents the amplitude of the signal. Since the human ear perceives high-frequency sounds less than low-frequency sounds, the vertical axis frequency of the Mel spectrogram uses the Mel scale, so that for audio with the same scale difference, the human ear's perception difference is basically the same, so the Mel spectrogram can be used to perform preliminary feature extraction on the audio. Of course, the Mel spectrogram can also be used to perform a discrete cosine transform (DCT for Discrete Cosine Transform) to further obtain the Mel cepstrum coefficients.

[0098] Taking the Mel feature as a Mel spectrogram as an example, in a specific implementation, when generating the Mel spectrogram corresponding to a panoramic audio clip, the audio signal of the panoramic audio clip can be first high-pass filtered to make the high and low frequency components of the audio signal relatively uniform, and then the audio signal is framed to obtain the relationship between the audio feature and time, and each frame needs to be windowed after framing; then, a Fast Fourier Transform (FFT) is performed on each frame, and the amplitude spectrum is taken for further calculation to obtain the power spectrum, and then the Mel filter group is applied to the power spectrum, and the amplitude is converted into logarithmic power, that is, the spectrogram of each frame is obtained, and the spectra of different frames are spliced ​​in the time dimension to obtain the Mel spectrogram corresponding to the panoramic audio clip.

[0099] In an embodiment of the present application, before extracting auditory features from a panoramic audio clip, the Mel features of the panoramic audio clip may be preprocessed to resize it into an input image with a preset aspect ratio. For example, the Mel features of the panoramic audio clip may be preprocessed into an input image with an aspect ratio of 1:2.

[0100] In the above implementation, since the Mel feature itself includes the time information of the audio, the auditory feature extracted based on the Mel feature of the panoramic audio clip can contain the time information, which is beneficial to improve the accuracy of the subsequent saliency prediction.

[0101] Considering that generating the Mel features corresponding to each panoramic audio clip is a relatively time-consuming process, in order to control the computational complexity and time cost, Figure 4 As shown, the aforementioned step S201 may include, when obtaining the panoramic video frame sequence and the panoramic audio clip sequence corresponding to the panoramic video to be processed:

[0102] S401, obtaining a panoramic video to be processed, and sequentially reading each video frame in the panoramic video to be processed to obtain a panoramic video frame sequence corresponding to the panoramic video to be processed.

[0103] S403, for each panoramic video frame in the panoramic video frame sequence, based on the playback time point of the panoramic video frame in the panoramic video to be processed, obtain the audio information corresponding to the time period between the first playback time point and the second playback time point from the panoramic video to be processed, and obtain the initial panoramic audio segment corresponding to the panoramic video frame.

[0104] Among them, the first playback time point is located before the playback time point of the panoramic video frame in the panoramic video to be processed and is separated from the playback time point by a first preset time length, and the second playback time point is located after the playback time point and is separated from the playback time point by a second preset time length.

[0105] The first preset duration and the second preset duration can be set based on actual needs, and the first preset duration and the second preset duration can be equal or unequal. Exemplarily, the first preset duration and the second preset duration can both be 0.5s, so that the panoramic audio segment is audio information of 1s in length.

[0106] It should be noted that, for the panoramic audio segments corresponding to the beginning and the end of the panoramic video to be processed, a time domain zero-padding operation may be performed on the corresponding audio, so that each panoramic audio segment contains audio information of the same duration.

[0107] S405: Convert the audio information of the initial panoramic audio segment corresponding to the panoramic video frame into a mono channel to obtain a panoramic audio segment corresponding to the panoramic video frame.

[0108] Specifically, for the audio information of the initial panoramic audio segment, the multi-channel audio is converted into mono audio to obtain a panoramic audio segment for subsequent auditory feature extraction, so that only mono Mel features can be used for auditory feature extraction, which can greatly reduce computational complexity and time expenditure.

[0109] S207, fusing the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audiovisual features corresponding to each panoramic video frame.

[0110] Specifically, the spatiotemporal visual features of each panoramic video frame and the auditory features of the corresponding panoramic audio clip may be linearly weighted summed to obtain the fused audiovisual features corresponding to each panoramic video frame, which may be calculated using the following formula (2):

[0111] f av =ω v ·f v +ω a ·f a (2)

[0112] Among them, f v Represents spatiotemporal visual features; f a Indicates auditory characteristics; f av Represents the fusion of audio-visual features; ω v With ω a Both represent weight coefficients and can be set based on actual experience.

[0113] Considering that spatiotemporal visual features and auditory features belong to two modal features and have large semantic differences, the fusion effect of simple linear fusion is limited. In order to further improve the accuracy of subsequent saliency prediction, in some exemplary embodiments, such as Figure 5 As shown, the above step S207 may include:

[0114] S501, for each of the spatiotemporal visual features corresponding to the panoramic video frame, channel-joining the spatiotemporal visual features and the auditory features of the panoramic audio clips corresponding to the panoramic video frame to obtain the channel-joining features corresponding to the panoramic video frame.

[0115] S503, performing convolution processing on the channel stitching features corresponding to the panoramic video frame to obtain fused audio-visual features corresponding to the panoramic video frame.

[0116] In a specific implementation, channel splicing can be shown as the following formula (3):

[0117]

[0118] in, Represents the channel concatenation feature; concatenate() represents the channel concatenation function.

[0119] Assuming the spatiotemporal visual feature f v The number of channels is 512, and the auditory features f a The number of channels is 512, then the channel splicing feature obtained after channel splicing based on the above formula (3) is The number of channels is 1024.

[0120] In a specific implementation, the convolution processing of the channel splicing features can be performed using the following formula (4):

[0121]

[0122] Among them, W av and β av It is the parameters and bias of the convolution kernel in the convolution process; represents the channel splicing feature; f av Represents the fusion of audio-visual features.

[0123] In the embodiment of the present application, the convolution kernel size of the convolution processing in the above step can be 1x1, so that the dimension of the channel splicing feature can be reduced to the dimension before the channel splicing. Taking the number of channels as 1024 as an example, the fused audio-visual features obtained after convolution processing based on the above formula (4) can be reduced to 512 channels.

[0124] In the above implementation, channel spatiotemporal visual features and auditory features are channel-split to obtain channel splicing features, and convolution processing is performed on the channel splicing features to obtain fused audio-visual features, which can realize the free interaction of spatiotemporal visual features and auditory features, thereby achieving a better audio-visual fusion effect.

[0125] S209, performing saliency prediction based on the fused audio-visual features corresponding to each panoramic video frame, and obtaining a saliency distribution map corresponding to each panoramic video frame.

[0126] Among them, the saliency distribution map corresponding to each panoramic video frame indicates the visual attention area.

[0127] The above-mentioned panoramic video processing method of the embodiment of the present application can be implemented based on a panoramic video processing network, which can be an encoder-decoder architecture, for example, a dual-stream structure based on a VGG (Visual Geometry Group) convolutional neural network.

[0128] Based on this, the embodiment of the present application may also include training the panoramic video processing network. Figure 6 The structural diagram of the panoramic video processing network shown in FIG. 1 is used to describe the training process of the panoramic video processing network in detail. Specifically, training the panoramic video processing network may include the following steps:

[0129] (1) Obtain a sample panoramic video frame sequence and a sample panoramic audio clip sequence corresponding to the sample panoramic video, each sample panoramic video frame corresponding to a reference saliency distribution map.

[0130] The reference saliency distribution map is the real saliency distribution map of the corresponding sample panoramic video frame. In a specific implementation, the original eye movement data can be collected through subjective experiments by the subjects, and then the focus point of the subjects on a certain video frame of the sample panoramic video can be extracted based on the collected original eye movement data through a clustering algorithm. Further processing can obtain the reference saliency distribution map corresponding to the video frame. Figure 7 Shown is an example of a sample panoramic video frame and its corresponding reference saliency distribution map.

[0131] The sample panoramic video frames in the sample panoramic video frame sequence are arranged according to the order in which they are played in the sample panoramic video.

[0132] The sample panoramic audio clip sequence is obtained based on the audio information of the sample panoramic video, and the sample panoramic audio clip sequence corresponds to the sample panoramic video frame sequence one by one, that is, each sample panoramic video frame corresponds to a sample panoramic audio clip. Specifically, each sample panoramic audio clip may include audio information of a preset duration before and after the corresponding sample panoramic video frame. For example, the preset duration may be 0.5s, that is, each sample panoramic audio clip may include audio information of 0.5s before the corresponding sample panoramic video frame and audio information of 0.5s after the sample panoramic video frame, so that each sample panoramic audio clip contains 1s of audio information.

[0133] It should be noted that, for the sample panoramic audio segments corresponding to the beginning and the end of the sample panoramic video, a time domain zero-padding operation may be performed on the corresponding audio so that each sample panoramic audio segment contains audio information of the same duration.

[0134] In the embodiment of the present application, the sample panoramic video frame may be a video frame in an ERP (Equirectangular Projection) format separated from the sample panoramic video.

[0135] (2) The sample panoramic video frames in the sample panoramic video frame sequence are sequentially input into the spatiotemporal visual feature extraction network of the initial panoramic video processing network to perform a sample feature extraction operation, and the sample feature extraction operation includes: performing a sample visual feature extraction on the current sample panoramic video frame, fusing the sample visual feature extraction result with the historical sample visual state to obtain the sample spatiotemporal visual feature of the current sample panoramic video frame, and updating the historical sample visual state based on the sample spatiotemporal visual feature of the current sample panoramic video frame.

[0136] In practical applications, before visual feature extraction is performed on the current sample panoramic video frame, the current sample panoramic video frame may be preprocessed to obtain an input sample image with a preset aspect ratio. For example, the current sample panoramic video frame in ERP format may be preprocessed into an input sample image with an aspect ratio of 1:2.

[0137] Among them, the spatiotemporal visual feature extraction network can be a convolutional neural network. When performing a sample feature extraction operation, the convolutional neural network can perform a convolution calculation on the input current sample panoramic video frame to obtain a convolution calculation result, and the convolution calculation result can be used as a sample visual feature extraction result.

[0138] Exemplarily, the fusion of the sample visual feature extraction result and the historical sample visual state can be performed on the basis of keeping the number of channels unchanged. For details, please refer to the aforementioned formula (1) of the embodiment of the present application, so as to integrate the time domain information into each sample visual feature extraction result in an exponential translation fusion manner to obtain the corresponding sample spatiotemporal visual features.

[0139] In another possible implementation, in order to achieve a better fusion effect, such as Figure 8The schematic diagram of the structure of another panoramic video processing network shown in the figure, the spatiotemporal visual feature extraction network may include a first convolution processing network and a time domain network, wherein the first convolution processing network is used to perform convolution processing on the current sample panoramic video frame to obtain a sample visual feature extraction result, and the current network parameters of the time domain network are used to calculate the weight coefficient of the sample visual feature extraction result, so as to achieve the weight coefficient α in the above formula (1) through learning. Based on this, the above step (2) may include:

[0140] Based on the first convolution processing network in the spatiotemporal visual feature extraction network, a convolution processing is performed on the current sample panoramic video frame, and the result of the convolution processing is used as the sample visual feature extraction result;

[0141] Obtaining current network parameters of the time domain network in the spatiotemporal visual feature extraction network, and determining weight coefficients of sample visual feature extraction results based on the current network parameters of the time domain network;

[0142] Based on the weight coefficient of the sample visual feature extraction result, the sample visual feature extraction result and the historical sample visual state are weighted and summed to obtain the sample spatiotemporal visual features of the current sample panoramic video frame; wherein the sum of the weight coefficient of the sample visual feature extraction result and the weight coefficient of the historical sample visual state is a preset value.

[0143] Specifically, the weighted summation process can be implemented by using the aforementioned formula (1). The weight coefficient α of the sample visual feature extraction result can be calculated based on the following formula (5):

[0144]

[0145] Among them, γ is the current network parameter of the time domain network, and the current network parameter is adjusted during the training process of the panoramic video processing network; sigmoid() is a nonlinear activation function.

[0146] (3) Based on the auditory feature extraction network of the initial panoramic video processing network, auditory features are extracted for each sample panoramic audio segment in the sample panoramic audio segment sequence to obtain sample auditory features of each sample panoramic audio segment.

[0147] Specifically, the auditory feature extraction network can be a convolutional neural network. When performing auditory feature extraction, a convolution calculation can be performed on the sample panoramic audio segment based on the convolutional neural network, and the result of the convolution calculation can be used as the sample auditory feature of the sample panoramic audio segment.

[0148] In some possible implementations, in order to enable the panoramic video processing network trained in the embodiment of the present application to extract auditory features in more complex scenes to improve the accuracy of saliency prediction in complex scenes, the above step (3) may include:

[0149] For each sample panoramic audio clip in the sample panoramic audio clip sequence, generating a sample Mel feature corresponding to the sample panoramic audio clip;

[0150] The sample Mel features corresponding to the sample panoramic audio clip are input into the auditory feature extraction network of the initial panoramic video processing network for auditory feature extraction to obtain the sample auditory features of the sample panoramic audio clip.

[0151] The sample Mel feature may be a Mel spectrogram or a Mel-scale Frequency Cepstral Coefficient (MFCC).

[0152] In practical applications, before extracting auditory features from a sample panoramic audio clip, the sample Mel features of the sample panoramic audio clip may be preprocessed to resize it into an input sample image with a preset aspect ratio. For example, the sample Mel features of the sample panoramic audio clip may be preprocessed into an input sample image with an aspect ratio of 1:2.

[0153] Considering that generating sample Mel features corresponding to each sample panoramic audio clip is a relatively time-consuming process, in order to control the computational complexity and time expenditure and improve the training efficiency, in an exemplary embodiment, the aforementioned step (1) may include, when obtaining the sample panoramic video frame sequence and the sample panoramic audio clip sequence corresponding to the sample panoramic video:

[0154] Obtain a sample panoramic video, read each video frame in the sample panoramic video in sequence, and obtain a sample panoramic video frame sequence corresponding to the sample panoramic video;

[0155] For each sample panoramic video frame in the sample panoramic video frame sequence, based on the sample playback time point of the sample panoramic video frame in the sample panoramic video, obtain the audio information corresponding to the time period between the first sample playback time point and the second sample playback time point from the sample panoramic video, and obtain the initial sample panoramic audio clip corresponding to the sample panoramic video frame; wherein the first sample playback time point is located before the sample playback time point and is separated from the sample playback time point by a first preset time length, and the second sample playback time point is located after the sample playback time point and is separated from the sample playback time point by a second preset time length;

[0156] The audio information of the initial sample panoramic audio segment corresponding to the sample panoramic video frame is converted into a mono channel to obtain a sample panoramic audio segment corresponding to the sample panoramic video frame.

[0157] Specifically, for the audio information of the initial sample panoramic audio clip, the multi-channel audio is converted into mono audio to obtain the sample panoramic audio clip for subsequent auditory feature extraction, so that only the mono sample Mel features can be used for auditory feature extraction, which can greatly reduce the computational complexity and time cost and improve the training efficiency.

[0158] (4) The sample spatiotemporal visual features of each sample panoramic video frame are fused with the sample auditory features of the corresponding sample panoramic audio clip to obtain the sample fused audio-visual features corresponding to each sample panoramic video frame.

[0159] Specifically, Figure 6 As shown, the sample spatiotemporal visual features of the sample panoramic video frame and the sample auditory features of the corresponding sample panoramic audio clip can be input into the audio-visual fusion layer of the initial sample panoramic video processing network for fusion processing to obtain the sample fused audio-visual features output by the layer.

[0160] Exemplarily, the aforementioned formula (2) in the embodiment of the present application may be used to fuse the sample spatiotemporal visual features with the corresponding sample auditory features.

[0161] Considering that the sample spatiotemporal visual features and the sample auditory features belong to two modal features and have large semantic differences, the fusion effect of simple linear fusion is limited. In order to further improve the accuracy of the trained panoramic video processing network, in some exemplary implementations, such as Fig. 9 As shown in the structural diagram of another panoramic video processing network, the above step (4) may include:

[0162] For each sample spatiotemporal visual feature corresponding to the sample panoramic video frame, channel-joining the sample spatiotemporal visual feature with the sample auditory feature of the sample panoramic audio clip corresponding to the sample panoramic video frame to obtain a sample channel-joined feature corresponding to the sample panoramic video frame;

[0163] The sample channel stitching features corresponding to the sample panoramic video frame are input into the second convolution processing network of the initial panoramic video processing network for convolution processing to obtain the sample fusion audio-visual features corresponding to the sample panoramic video frame.

[0164] In a specific implementation, the sample spatiotemporal visual features and the corresponding sample auditory features can be input into Fig. 9The channel splicing layer shown performs channel splicing to obtain the sample channel splicing features output by the layer, and then the sample channel splicing features can be input into the second convolution processing network for convolution processing. Wherein, channel splicing can be implemented by the aforementioned formula (3) of the embodiment of the present application.

[0165] The convolution kernel size of the second convolution processing network can be 1x1, so that the dimension of the sample channel splicing feature can be reduced to the dimension before channel splicing. For the specific convolution processing of the sample channel splicing feature, please refer to the aforementioned formula (4) of the embodiment of the present application.

[0166] In the above implementation, the sample spatiotemporal visual features and the sample auditory features are channel-spliced ​​to obtain sample channel-spliced ​​features, and the sample channel-spliced ​​features are convolved using a second convolution processing network to obtain sample fused audio-visual features. This can achieve free interaction between the sample spatiotemporal visual features and the sample auditory features, thereby achieving a better audio-visual fusion effect.

[0167] (5) Based on the saliency prediction network of the initial panoramic video processing network, saliency prediction is performed on the sample fused audio-visual features corresponding to each sample panoramic video frame to obtain a predicted saliency distribution map corresponding to each sample panoramic video frame.

[0168] Among them, the saliency prediction network can be a decoding network, and the sample fused audio-visual features corresponding to each sample panoramic video frame can be input into the decoding network for decoding processing, so as to obtain the predicted saliency distribution map corresponding to each sample panoramic video frame as output.

[0169] (6) Based on the difference between the predicted saliency distribution map corresponding to each sample panoramic video frame and the corresponding reference saliency distribution map, the network parameters of the initial panoramic video processing network are adjusted until the preset training end condition is met, thereby obtaining the panoramic video processing network.

[0170] Specifically, a preset loss function can be used to determine the loss value in combination with the difference between the predicted saliency distribution map corresponding to each sample panoramic video frame and the corresponding reference saliency distribution map, and the network parameters of the initial panoramic video processing network can be reversely adjusted based on the loss value, and training can be continued based on the adjusted network parameters until the preset training end condition is met to end the training, thereby obtaining a panoramic video processing network, and the network parameters of the panoramic video processing network are the network parameters at the end of the training. The network parameters adjusted during the training process include all network parameters in the initial panoramic video processing network.

[0171] Specifically, the preset training end condition may be that the number of iterations reaches a preset iteration number threshold, or that the loss value reaches a preset loss threshold, or that the difference between the loss values ​​of two adjacent iterations is less than a preset loss difference.

[0172] The preset loss function may be a binary cross entropy loss function, based on which the loss value may be calculated using the following formula (6):

[0173]

[0174] Where N is the number of image pixels; S pd is the predicted significance distribution map; S gt is the reference significance distribution map.

[0175] In practical applications, in order to improve training efficiency, the Adam optimizer can be used, with a setting of 10 -4 The regularized weight decay is implemented and different learning rates are assigned to each module.

[0176] A panoramic video processing network can be obtained through training in the above steps, and then the panoramic video processing network can be used to implement the aforementioned panoramic video processing method of the embodiment of the present application, which can not only reduce the computing resource consumption and time overhead of saliency prediction, but also improve the accuracy of saliency prediction in complex sound scenes, and can also track salient objects well in scenes with moving sounds, giving more accurate saliency prediction results.

[0177] After the panoramic video processing network is trained, the panoramic video processing network can be used to implement the panoramic video processing method of the embodiment of the present application. Then, the aforementioned step S203, when extracting visual features of the current panoramic video frame and fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, can include:

[0178] Based on the first convolution processing network in the panoramic video processing network, convolution processing is performed on the current panoramic video frame, and the result of the convolution processing is used as the visual feature extraction result;

[0179] Obtaining network parameters of a time domain network in a panoramic video processing network, and determining weight coefficients of visual feature extraction results based on the network parameters;

[0180] Based on the weight coefficient of the visual feature extraction result, the visual feature extraction result and the historical visual state are weighted and summed to obtain the spatiotemporal visual features of the current panoramic video frame; wherein the sum of the weight coefficient of the visual feature extraction result and the weight coefficient of the historical visual state is a preset value.

[0181] Among them, the weight coefficient of the visual feature extraction result determined based on the network parameters of the time domain network can be determined by the aforementioned formula (5), and the weighted summation process can be performed by the aforementioned formula (1), which will not be repeated here.

[0182] In order to facilitate understanding of the technical solution of the embodiment of the present application, Fig.10 The panoramic video processing method of the present application embodiment is described by taking the structure of the panoramic video processing network shown in FIG. Fig.10 The output size of each layer of the network is shown in .

[0183] It should be noted that the data set used to train the panoramic video processing network may include 162 sample panoramic videos and their corresponding attention annotations (i.e., reference saliency distribution maps). These sample panoramic videos can be divided into three categories according to the number of salient objects and sound sources in the video: (1) there is only one salient object and the sound source is the salient object; (2) there are multiple salient objects, and the sound source is one of them; (3) there are multiple salient objects and multiple sound sources. Among them, salient objects mainly refer to people and moving objects. When using this data set, the embodiment of the present application divides it into 142 training videos and 20 test videos. The model reached the optimal value when the 25th epoch was completed on the training set. At this time, γ≈-2.8393 in the time domain network, and the weight coefficient α≈0.0552 can be obtained by calculation.

[0184] like Fig.10 As shown in the figure, the panoramic video processing network follows the encoder-decoder architecture, takes panoramic video frames and corresponding panoramic audio clips as input, uses a dual-stream structure based on the VGG convolutional neural network to extract the visual and auditory features of the video respectively, and predicts the saliency distribution map from the fused multimodal features.

[0185] For visual feature extraction, the VGG-16 convolutional neural network is used as the video feature extractor. The final maximum pooling layer and fully connected layer are removed. This part of the network processes visual information directly in the ERP format. The visual feature extractor reads in one video frame at a time and preprocesses the image. During the processing, the ERP panoramic image aspect ratio of 1:2 is maintained, so that the image input to the VGG network has a size of 240×480×3, where "240" and "480" are the height and width of the processed image, respectively, and "3" corresponds to the RGB channels of the image. After extraction by the VGG-16 network, a feature map of size 15×30×512 can be obtained.

[0186] In order to consider the temporal information of the video, a temporal network is set at the center of the codec network. Specifically, the feature maps extracted by VGG-16 are first Then apply a convolution layer to perform convolution calculation to obtain the current convolution result. On the basis of keeping the number of channels of the feature map unchanged, the above formula (1) is used to make the final output result of the time domain network (i.e., the spatiotemporal visual feature) the weighted average of the current convolution result and the previous historical visual state, and the final output result of the time domain network is used as the historical visual state at the next moment. Among them, the weight coefficient used in the weighted average processing is calculated based on the network parameters of the time domain network combined with the above formula (5).

[0187] For auditory feature extraction, generate the Mel spectrogram of the panoramic audio clip, process the size of the Mel spectrogram to 240×480×3, and use the VGG-16 network to extract the feature map f with a size of 15×30×512 a , which serves as the corresponding auditory feature.

[0188] For audio-visual feature fusion, f v and f a The two features are concatenated to obtain a feature map with 1024 channels. Then, the dimension is reduced to 512 channels through a convolutional layer with a kernel size of 1×1, and a fused feature map f with a size of 15×30×512 is obtained. av , as the final input of the decoder.

[0189] For saliency distribution map prediction, in the decoder part, a network architecture with the same structure as the encoder part but in the opposite order can be used, and the maximum pooling layer is replaced by the nearest upsampling layer. The resolution of the predicted saliency distribution map finally reaches 240×480.

[0190] It is understandable that the VGG-16 architecture in the above-mentioned visual and auditory feature extraction network can also be a convolutional neural network architecture such as VGG-19. In the decoder part, a deconvolution layer can also be used to replace the nearest neighbor method layer in the decoder network.

[0191] Corresponding to the panoramic video processing methods provided in the above-mentioned embodiments, an embodiment of the present application also provides a panoramic video processing device. Since the panoramic video processing device provided in the embodiment of the present application corresponds to the panoramic video processing methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned panoramic video processing methods are also applicable to the panoramic video processing device provided in this embodiment and will not be described in detail in this embodiment.

[0192] See also Fig.11 , which is a schematic diagram of the structure of a panoramic video processing device provided by an embodiment of the present invention, the device has the function of implementing the panoramic video processing method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Fig.11As shown, the panoramic video processing device 1100 may include:

[0193] The frame sequence acquisition module 1110 is used to acquire a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed;

[0194] The spatiotemporal visual feature extraction module 1120 is used to perform a feature extraction operation on the panoramic video frames in the panoramic video frame sequence in sequence, wherein the feature extraction operation includes: extracting visual features on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame;

[0195] The auditory feature extraction module 1130 is used to extract auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment;

[0196] The audio-visual fusion module 1140 is used to fuse the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audio-visual features corresponding to each panoramic video frame;

[0197] The saliency prediction module 1150 is used to perform saliency prediction based on the fused audio-visual features corresponding to each panoramic video frame, and obtain a saliency distribution map corresponding to each panoramic video frame.

[0198] In an exemplary embodiment, the audio-visual fusion module 1140 includes:

[0199] A channel stitching module is used to perform channel stitching on the spatiotemporal visual features corresponding to each panoramic video frame and the auditory features of the panoramic audio clip corresponding to the panoramic video frame to obtain the channel stitching features corresponding to the panoramic video frame;

[0200] The first convolution processing module is used to perform convolution processing on the channel stitching features corresponding to the panoramic video frame to obtain the fused audio-visual features corresponding to the panoramic video frame.

[0201] In an exemplary embodiment, the auditory feature extraction module 1130 includes:

[0202] A mel feature generation module, configured to generate a mel feature corresponding to each panoramic audio segment in the panoramic audio segment sequence;

[0203] The auditory feature extraction submodule is used to extract auditory features from the Mel features corresponding to the panoramic audio segment to obtain the auditory features of the panoramic audio segment.

[0204] In an exemplary embodiment, the frame sequence acquisition module 1110 includes:

[0205] A video frame sequence determination module is used to obtain a panoramic video to be processed, read each video frame in the panoramic video to be processed in sequence, and obtain a panoramic video frame sequence corresponding to the panoramic video to be processed;

[0206] an initial audio segment determination module, for each panoramic video frame in the panoramic video frame sequence, based on the playback time point of the panoramic video frame in the panoramic video to be processed, obtaining audio information corresponding to a time period between a first playback time point and a second playback time point from the panoramic video to be processed, to obtain an initial panoramic audio segment corresponding to the panoramic video frame; the first playback time point is located before the playback time point and is separated from the playback time point by a first preset time length, and the second playback time point is located after the playback time point and is separated from the playback time point by a second preset time length;

[0207] The audio segment determination module is used to convert the audio information of the initial panoramic audio segment corresponding to the panoramic video frame into mono, so as to obtain the panoramic audio segment corresponding to the panoramic video frame.

[0208] In an exemplary embodiment, the apparatus further comprises:

[0209] A sample acquisition module, used to acquire a sample panoramic video frame sequence and a sample panoramic audio clip sequence corresponding to a sample panoramic video, each sample panoramic video frame corresponding to a reference saliency distribution map;

[0210] A sample spatiotemporal visual feature extraction module is used to sequentially input the sample panoramic video frames in the sample panoramic video frame sequence into the spatiotemporal visual feature extraction network of the initial panoramic video processing network to perform a sample feature extraction operation, wherein the sample feature extraction operation includes: performing sample visual feature extraction on the current sample panoramic video frame, fusing the sample visual feature extraction result with the historical sample visual state to obtain the sample spatiotemporal visual feature of the current sample panoramic video frame, and updating the historical sample visual state based on the sample spatiotemporal visual feature of the current sample panoramic video frame;

[0211] A sample auditory feature extraction module, configured to extract auditory features from each sample panoramic audio segment in the sample panoramic audio segment sequence based on the auditory feature extraction network of the initial panoramic video processing network, so as to obtain sample auditory features of each sample panoramic audio segment;

[0212] A sample audio-visual fusion module is used to fuse the sample spatiotemporal visual features of each sample panoramic video frame with the sample auditory features of the corresponding sample panoramic audio clip to obtain the sample fused audio-visual features corresponding to each sample panoramic video frame;

[0213] A prediction module, configured to perform saliency prediction on the sample fused audiovisual features corresponding to each sample panoramic video frame based on the saliency prediction network of the initial panoramic video processing network, and obtain a predicted saliency distribution map corresponding to each sample panoramic video frame;

[0214] The training module is used to adjust the network parameters of the initial panoramic video processing network based on the difference between the predicted saliency distribution map corresponding to each sample panoramic video frame and the corresponding reference saliency distribution map until the preset training end condition is met to obtain the panoramic video processing network.

[0215] In an exemplary embodiment, the sample spatiotemporal visual feature extraction module includes:

[0216] A second convolution processing module is used to perform convolution processing on the current sample panoramic video frame based on the first convolution processing network in the spatiotemporal visual feature extraction network, and use the result of the convolution processing as the sample visual feature extraction result;

[0217] A first weight system determination module is used to obtain the current network parameters of the time domain network in the spatiotemporal visual feature extraction network, and determine the weight coefficient of the sample visual feature extraction result based on the current network parameters of the time domain network;

[0218] The first weighted processing module is used to perform weighted summation processing on the sample visual feature extraction result and the historical sample visual state based on the weight coefficient of the sample visual feature extraction result to obtain the sample spatiotemporal visual features of the current sample panoramic video frame; wherein the sum of the weight coefficient of the sample visual feature extraction result and the weight coefficient of the historical sample visual state is a preset value.

[0219] In an exemplary embodiment, the sample audio-visual fusion module includes:

[0220] A sample channel stitching module is used to perform channel stitching on the sample spatiotemporal visual features corresponding to each sample panoramic video frame and the sample auditory features of the sample panoramic audio clip corresponding to the sample panoramic video frame to obtain the sample channel stitching features corresponding to the sample panoramic video frame;

[0221] The third convolution processing module is used to input the sample channel stitching features corresponding to the sample panoramic video frame into the second convolution processing network of the initial panoramic video processing network for convolution processing to obtain the sample fusion audio-visual features corresponding to the sample panoramic video frame.

[0222] In an exemplary embodiment, the sample auditory feature extraction module includes:

[0223] A sample Mel feature generation module, used for generating, for each sample panoramic audio segment in the sample panoramic audio segment sequence, a sample Mel feature corresponding to the sample panoramic audio segment;

[0224] The sample auditory feature extraction submodule is used to input the sample Mel features corresponding to the sample panoramic audio clip into the auditory feature extraction network of the initial panoramic video processing network for auditory feature extraction to obtain the sample auditory features of the sample panoramic audio clip.

[0225] In an exemplary embodiment, the sample acquisition module includes:

[0226] A sample video frame determination module is used to obtain a sample panoramic video, read each video frame in the sample panoramic video in sequence, and obtain a sample panoramic video frame sequence corresponding to the sample panoramic video;

[0227] An initial sample audio segment determination module is used to obtain, for each sample panoramic video frame in the sample panoramic video frame sequence, audio information corresponding to a time period between a first sample playback time point and a second sample playback time point from the sample panoramic video based on a sample playback time point of the sample panoramic video frame in the sample panoramic video, so as to obtain an initial sample panoramic audio segment corresponding to the sample panoramic video frame; the first sample playback time point is located before the sample playback time point and is separated from the sample playback time point by a first preset time length, and the second sample playback time point is located after the sample playback time point and is separated from the sample playback time point by a second preset time length;

[0228] The sample panoramic audio segment determination module is used to convert the audio information of the initial sample panoramic audio segment corresponding to the sample panoramic video frame into mono, so as to obtain the sample panoramic audio segment corresponding to the sample panoramic video frame.

[0229] In an exemplary embodiment, when the spatiotemporal visual feature extraction module extracts visual features of the current panoramic video frame and fuses the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, it is specifically used to: perform convolution processing on the current panoramic video frame based on the first convolution processing network in the panoramic video processing network, and use the result of the convolution processing as the visual feature extraction result; obtain the network parameters of the time domain network in the panoramic video processing network, and determine the weight coefficient of the visual feature extraction result based on the network parameters; based on the weight coefficient of the visual feature extraction result, perform weighted summation processing on the visual feature extraction result and the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame; wherein the sum of the weight coefficient of the visual feature extraction result and the weight coefficient of the historical visual state is the preset value.

[0230] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0231] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any panoramic video processing method provided in the above method embodiments.

[0232] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0233] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a server or a similar computing device, that is, the above electronic device can include a computer terminal, a server or a similar computing device. Taking running on a server as an example, Fig.12 is a hardware structure block diagram of an electronic device running a panoramic video processing method provided by an embodiment of the present invention, such as Fig.12As shown, the server 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1210 (the processor 1210 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1230 for storing data, and one or more storage media 1220 (such as one or more mass storage devices) for storing application programs 1223 or data 1222. Among them, the memory 1230 and the storage medium 1220 can be short-term storage or permanent storage. The program stored in the storage medium 1220 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1210 can be configured to communicate with the storage medium 1220 and execute a series of instruction operations in the storage medium 1220 on the server 1200. The server 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1240, and / or one or more operating systems 1221, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0234] The input / output interface 1240 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 1200. In one example, the input / output interface 1240 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 1240 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0235] It can be understood by those skilled in the art that Fig.12 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.12 More or fewer components as shown, or with Fig.12 Different configurations are shown.

[0236] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a panoramic video processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any panoramic video processing method provided by the above method embodiments.

[0237] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device performs any one of the panoramic video processing methods provided in the above method embodiments.

[0238] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0239] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0240] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0241] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0242] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A panoramic video processing method, characterized in that: The method comprises: Obtain a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed; Performing a feature extraction operation on the panoramic video frames in the panoramic video frame sequence in sequence, the feature extraction operation comprising: extracting visual features on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame; Extracting auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment; Fusing the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audiovisual features corresponding to each panoramic video frame; Based on the fused audio-visual features corresponding to each of the panoramic video frames, saliency prediction is performed respectively to obtain a saliency distribution map corresponding to each of the panoramic video frames.

2. The method according to claim 1, characterized in that The fusing the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain the fused audiovisual features corresponding to each panoramic video frame includes: For each of the spatiotemporal visual features corresponding to the panoramic video frame, channel-joining the spatiotemporal visual features with the auditory features of the panoramic audio clips corresponding to the panoramic video frame to obtain the channel-joining features corresponding to the panoramic video frame; Convolution processing is performed on the channel stitching features corresponding to the panoramic video frame to obtain fused audio-visual features corresponding to the panoramic video frame.

3. The method according to claim 1, characterized in that: The extracting of auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain the auditory features of each panoramic audio segment includes: For each panoramic audio segment in the panoramic audio segment sequence, generating a Mel feature corresponding to the panoramic audio segment; Performing auditory feature extraction on the Mel features corresponding to the panoramic audio segment to obtain the auditory features of the panoramic audio segment.

4. The method according to claim 3, characterized in that The step of obtaining a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed includes: Acquire a panoramic video to be processed, and read each video frame in the panoramic video to be processed in sequence to obtain a panoramic video frame sequence corresponding to the panoramic video to be processed; For each panoramic video frame in the panoramic video frame sequence, based on the playback time point of the panoramic video frame in the panoramic video to be processed, audio information corresponding to a time period between a first playback time point and a second playback time point is acquired from the panoramic video to be processed to obtain an initial panoramic audio segment corresponding to the panoramic video frame; the first playback time point is located before the playback time point and is separated from the playback time point by a first preset time length, and the second playback time point is located after the playback time point and is separated from the playback time point by a second preset time length; The audio information of the initial panoramic audio segment corresponding to the panoramic video frame is converted into a mono channel to obtain a panoramic audio segment corresponding to the panoramic video frame.

5. The method according to any one of claims 1 to 4, characterized in that The panoramic video processing method is implemented based on a panoramic video processing network, and the training of the panoramic video processing network includes: Obtaining a sample panoramic video frame sequence and a sample panoramic audio clip sequence corresponding to the sample panoramic video, each sample panoramic video frame corresponding to a reference saliency distribution map; The sample panoramic video frames in the sample panoramic video frame sequence are sequentially input into the spatiotemporal visual feature extraction network of the initial panoramic video processing network to perform a sample feature extraction operation, wherein the sample feature extraction operation includes: performing sample visual feature extraction on the current sample panoramic video frame, fusing the sample visual feature extraction result with the historical sample visual state to obtain the sample spatiotemporal visual feature of the current sample panoramic video frame, and updating the historical sample visual state based on the sample spatiotemporal visual feature of the current sample panoramic video frame; Based on the auditory feature extraction network of the initial panoramic video processing network, respectively extracting auditory features from each sample panoramic audio segment in the sample panoramic audio segment sequence to obtain sample auditory features of each sample panoramic audio segment; Fusing the sample spatiotemporal visual features of each of the sample panoramic video frames with the sample auditory features of the corresponding sample panoramic audio clips to obtain sample fused audiovisual features corresponding to each of the sample panoramic video frames; Based on the saliency prediction network of the initial panoramic video processing network, respectively predicting the saliency of the sample fused audiovisual features corresponding to each of the sample panoramic video frames, to obtain a predicted saliency distribution map corresponding to each of the sample panoramic video frames; Based on the difference between the predicted saliency distribution map corresponding to each of the sample panoramic video frames and the corresponding reference saliency distribution map, the network parameters of the initial panoramic video processing network are adjusted until a preset training end condition is met to obtain the panoramic video processing network.

6. The method according to claim 5, characterized in that The extracting sample visual features of the current sample panoramic video frame and fusing the sample visual feature extraction result with the historical sample visual state to obtain the sample spatiotemporal visual features of the current sample panoramic video frame includes: Performing convolution processing on the current sample panoramic video frame based on the first convolution processing network in the spatiotemporal visual feature extraction network, and using the result of the convolution processing as the sample visual feature extraction result; Acquiring current network parameters of a time domain network in the spatiotemporal visual feature extraction network, and determining a weight coefficient of the sample visual feature extraction result based on the current network parameters of the time domain network; Based on the weight coefficient of the sample visual feature extraction result, the sample visual feature extraction result and the historical sample visual state are weighted and summed to obtain the sample spatiotemporal visual features of the current sample panoramic video frame; wherein the sum of the weight coefficient of the sample visual feature extraction result and the weight coefficient of the historical sample visual state is a preset value.

7. The method according to claim 5, characterized in that The fusing the sample spatiotemporal visual features of each of the sample panoramic video frames with the sample auditory features of the corresponding sample panoramic audio clips to obtain the sample fused audiovisual features corresponding to each of the sample panoramic video frames includes: For each sample spatiotemporal visual feature corresponding to the sample panoramic video frame, channel-joining the sample spatiotemporal visual feature with the sample auditory feature of the sample panoramic audio clip corresponding to the sample panoramic video frame to obtain a sample channel-joined feature corresponding to the sample panoramic video frame; The sample channel stitching features corresponding to the sample panoramic video frame are input into the second convolution processing network of the initial panoramic video processing network for convolution processing to obtain the sample fusion audio-visual features corresponding to the sample panoramic video frame.

8. The method according to claim 5, characterized in that The auditory feature extraction network based on the initial panoramic video processing network extracts auditory features from each sample panoramic audio segment in the sample panoramic audio segment sequence, and obtains sample auditory features of each sample panoramic audio segment, including: For each sample panoramic audio segment in the sequence of sample panoramic audio segments, generating a sample Mel feature corresponding to the sample panoramic audio segment; The sample Mel features corresponding to the sample panoramic audio clip are input into the auditory feature extraction network of the initial panoramic video processing network for auditory feature extraction to obtain the sample auditory features of the sample panoramic audio clip.

9. The method according to claim 8, characterized in that The step of obtaining a sample panoramic video frame sequence and a sample panoramic audio clip sequence corresponding to the sample panoramic video includes: Acquire a sample panoramic video, and read each video frame in the sample panoramic video in sequence to obtain a sample panoramic video frame sequence corresponding to the sample panoramic video; For each sample panoramic video frame in the sample panoramic video frame sequence, based on the sample playback time point of the sample panoramic video frame in the sample panoramic video, audio information corresponding to the time period between a first sample playback time point and a second sample playback time point is obtained from the sample panoramic video to obtain an initial sample panoramic audio clip corresponding to the sample panoramic video frame; the first sample playback time point is located before the sample playback time point and is separated from the sample playback time point by a first preset time length, and the second sample playback time point is located after the sample playback time point and is separated from the sample playback time point by a second preset time length; The audio information of the initial sample panoramic audio segment corresponding to the sample panoramic video frame is converted into a mono channel to obtain a sample panoramic audio segment corresponding to the sample panoramic video frame.

10. The method according to claim 6, characterized in that The extracting visual features of the current panoramic video frame and fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame includes: Performing convolution processing on the current panoramic video frame based on the first convolution processing network in the panoramic video processing network, and using the result of the convolution processing as a visual feature extraction result; Acquire network parameters of a time domain network in the panoramic video processing network, and determine a weight coefficient of the visual feature extraction result based on the network parameters; Based on the weight coefficient of the visual feature extraction result, the visual feature extraction result and the historical visual state are weighted and summed to obtain the spatiotemporal visual features of the current panoramic video frame; wherein the sum of the weight coefficient of the visual feature extraction result and the weight coefficient of the historical visual state is the preset value.

11. A panoramic video processing device, characterized in that: The device comprises: A frame sequence acquisition module, used to acquire a panoramic video frame sequence and a panoramic audio clip sequence corresponding to the panoramic video to be processed; a spatiotemporal visual feature extraction module, configured to sequentially perform feature extraction operations on the panoramic video frames in the panoramic video frame sequence, wherein the feature extraction operations include: extracting visual features on the current panoramic video frame, fusing the video feature extraction result with the historical visual state to obtain the spatiotemporal visual features of the current panoramic video frame, and updating the historical visual state based on the spatiotemporal visual features of the current panoramic video frame; an auditory feature extraction module, configured to extract auditory features from each panoramic audio segment in the panoramic audio segment sequence to obtain auditory features of each panoramic audio segment; An audio-visual fusion module, used to fuse the spatiotemporal visual features of each panoramic video frame with the auditory features of the corresponding panoramic audio clip to obtain fused audio-visual features corresponding to each panoramic video frame; The saliency prediction module is used to perform saliency prediction based on the fused audio-visual features corresponding to each panoramic video frame to obtain a saliency distribution map corresponding to each panoramic video frame.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the panoramic video processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the panoramic video processing method according to any one of claims 1 to 10.

Images