Video processing method, device, electronic device and storage medium

By predicting the user's subsequent viewing perspective and differentiating the code to transmit panoramic videos, the existing panoramic video transmission problems are solved, and the user experience and transmission efficiency are improved.

CN114979652BActive Publication Date: 2024-11-12DOUYIN VISION CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210553339.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-20
Publication Date
2024-11-12
Estimated Expiration
2042-05-20

AI Technical Summary

Technical Problem

The existing panoramic video encoding transmission methods are redundant, resulting in slow video transmission speed, poor user viewing experience, and unable to see the area of ​​interest in real time.

Method used

By determining the historical perspective significant diagram and the video content significant diagram of the historical time slice, input it into the pre-trained target perspective prediction model, predict the user's subsequent viewing perspective, and differentiate the code to transmit videos based on the prediction results to improve the user experience.

Benefits of technology

It realizes the differentiated code transmission of videos based on the user's subsequent viewing perspective, which improves the user's viewing experience and video transmission speed, and improves the prediction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114979652B_ABST
    Figure CN114979652B_ABST
Patent Text Reader

Abstract

The disclosed embodiments provide a video processing method, device, electronic device and storage medium, the method comprising: determining a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency image; wherein the historical time slice is the previous time slice of the predicted time slice; inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice; and determining a target video within the predicted time slice based on the predicted video content saliency map. The technical solution of the disclosed embodiments realizes the prediction of the user's subsequent viewing perspective, facilitates providing the user with a corresponding target video based on the prediction result, and improves the user's viewing experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a video processing method, device, electronic device, and storage medium. Background Art

[0002] At present, virtual reality technology and its related applications are developing very rapidly. As an important component of virtual reality technology, the transmission and processing of panoramic video plays a very critical role in virtual reality applications.

[0003] In the prior art, when transmitting panoramic videos in a traditional video encoding method, it is necessary to encode the images of each part of the panoramic video with the same or similar quality to generate a code stream, so as to transmit the video to the user end. Therefore, the existing panoramic video encoding transmission method has a large redundancy, the video transmission speed is slow, and the user's viewing experience is poor. Summary of the invention

[0004] The present disclosure provides a video processing method, device, electronic device and storage medium, which realize the prediction of the user's subsequent viewing angle, facilitate the provision of corresponding target videos to the user according to the prediction results, and improve the user's viewing experience.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map; wherein the historical time slice is a previous time slice of the predicted time slice;

[0007] Inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice;

[0008] Based on the predicted video content saliency map, a target video within the predicted time slice is determined.

[0009] In a second aspect, the present disclosure also provides a video processing device, including:

[0010] A historical perspective saliency map determination module, used to determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency image; wherein the historical time slice is a previous time slice of the predicted time slice;

[0011] A predicted video content saliency map determination module is used to input the historical view saliency map, the historical video content saliency map and the current view saliency map into a pre-trained target view prediction model to obtain a predicted video content saliency map of the predicted time slice;

[0012] The target video determination module is used to determine the target video within the predicted time slice based on the predicted video content saliency map.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in any one of the embodiments of the present disclosure.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the video processing method as described in any one of the embodiments of the present disclosure.

[0018] The technical solution of the disclosed embodiment determines a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map. Furthermore, the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map are input into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice. Finally, based on the predicted video content saliency map, the target video within the predicted time slice is determined, and a machine learning model is used to predict the user's subsequent viewing perspective, so as to provide the user with a corresponding target video based on the prediction results, thereby improving the user's viewing experience. At the same time, the influence of the video content on the user's viewing perspective is taken into account during the prediction process, thereby improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A schematic diagram of a video processing method provided in the first embodiment of the present disclosure;

[0021] Figure 2A schematic diagram of determining a predicted perspective saliency map based on a target perspective prediction model provided in the first embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a video processing flow provided in Embodiment 1 of the present disclosure;

[0023] Figure 4 A schematic diagram of the structure of a video processing device provided in Embodiment 2 of the present disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] Before introducing the technical solution, an example of the application scenario of the embodiment of the present disclosure can be first explained. For example, when a user uses an augmented reality (AR) device or a virtual reality (VR) device to watch a panoramic video transmitted by a server, usually only a part of the panoramic video can be seen. If the server encodes each part of the panoramic video with the same or similar quality, the transmitted panoramic video will occupy a large bandwidth and the data transmission speed will be slow, which will result in the user being unable to see the area of ​​interest in the panoramic video in real time during the subsequent viewing process. At this time, based on the solution of the embodiment of the present disclosure, the server or the client can determine the distribution of the user's perspective in the predicted time slice according to the historical video content watched by the user, and then determine the corresponding target video based on the predicted perspective distribution. It can be understood that the pictures of each part of the panoramic video are transmitted to the client at a differentiated bit rate, so that the user can smoothly watch the part of the panoramic video that he is interested in in the subsequent process, thereby ensuring the user's viewing experience.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a video processing method provided in the first embodiment of the present disclosure. The embodiment of the present disclosure is applicable to a situation in which, when a user is watching a panoramic video, the viewing angle of the user in the next time period is predicted, so as to provide the user with a corresponding target video. The method can be executed by a video processing device, which can be implemented in the form of software and / or hardware, and optionally, can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0033] like Figure 1 As shown, the method includes:

[0034] S110: Determine a historical perspective salient map of a historical time slice and a historical video content salient map corresponding to the historical perspective salient image.

[0035] When a user uses an AR device or a VR device to watch a panoramic video, it is first necessary to determine the time slice corresponding to the panoramic video. Among them, a panoramic video is a video obtained by shooting a 360° video with a 3D camera. When a user uses a corresponding device to watch a panoramic video, the picture of the corresponding area in the panoramic video can be viewed by changing the viewing angle. In this embodiment, for the panoramic video transmitted to the user end, the corresponding time slice can be determined according to the length of the predicted time or the pre-set division rule. At the same time, for each time slice, there is at least one video frame corresponding to it. For example, when the predicted time length is one second, the time slice corresponds to a panoramic video frame within one second of the panoramic video. On this basis, when predicting the user's subsequent viewing angle, the predicted time period is the predicted time slice of the panoramic video. At the same time, relative to the predicted time period, the previous time period is the historical time slice of the panoramic video.

[0036] For example, when the predicted time length is one second, and the user watches a panoramic video through an AR device or a VR device, for the current moment, the second before the moment is the historical time slice, and accordingly, the server or the user can predict the user's perspective within one second after the moment, and at this time, the next second is the predicted time slice; when the predicted time length is ten seconds, for the current moment, the ten seconds before the moment is the historical time slice, and accordingly, the server or the user can predict the user's perspective within ten seconds after the moment, and at this time, the next ten seconds is the predicted time slice. Of course, in actual application, the length of the predicted time can be set according to actual conditions, and accordingly, the length of the time slice will also be adjusted according to the actual set prediction time length, which will not be elaborated in the embodiments of the present disclosure.

[0037] In this embodiment, after determining the historical time slice, it is also necessary to determine the historical perspective saliency map. In machine vision, saliency is a mode of image partitioning, and a saliency map is an image that shows the uniqueness of each pixel. It can be understood that the goal of the saliency map is to simplify or change the representation of a general image into a style that is easier to analyze. For example, for the entire picture contained in a panoramic video, the pixels corresponding to a certain area can be represented in red, and the pixels corresponding to another area can be represented in blue, or the pixels corresponding to a certain area can be assigned a label, and the pixels corresponding to another area can be assigned another label.

[0038] On this basis, for each panoramic video frame in the historical time slice, it includes a historical perspective saliency map and a historical video content saliency map corresponding to the historical perspective saliency map, wherein both the historical video content saliency map and the historical perspective saliency map are panoramic maps. Specifically, the historical perspective saliency map is a panoramic picture that displays the area that the user has watched in red or highlighted colors on the basis of at least one panoramic video frame corresponding to the historical time slice, and displays the area that the user has not watched in blue or dim colors; the historical video content saliency map is a panoramic picture that displays the area in the video screen that is easy to interest the user in red or other highlighted colors on the basis of at least one panoramic video frame corresponding to the historical time slice, and displays the area in the video screen that is not easy to interest the user in blue or dim colors.

[0039] It can be understood that the historical perspective saliency map is a fused image. Specifically, since there are usually multiple panoramic video frames corresponding to the historical time slice, after determining the corresponding perspective saliency map for each panoramic video frame, the multiple perspective saliency maps can be fused to obtain the historical perspective saliency map corresponding to the historical time slice; similarly, the historical video content saliency map can also be obtained by fusing the video content saliency maps corresponding to multiple panoramic video frames, which will not be described in detail in the present disclosed embodiment. It should be noted that since the area viewed by the user may not be consistent with the area determined by the server or client to be likely to interest the user, there may be differences between the salient areas in the historical perspective saliency map and the salient areas in the historical video screen. The process of determining the above two saliency maps is described below.

[0040] Optionally, when the last video frame in the historical time slice is obtained, the historical perspective saliency map and the historical video content saliency map of the historical time slice are determined; or, when the historical video frame in the historical time slice is obtained, the historical perspective saliency map and the historical video content saliency map corresponding to the historical time slice are updated based on the historical video frame.

[0041] Continuing with the above example, when the user has watched part of the pictures in all 30 video frames corresponding to the first time slice, in order to predict the area that the user may watch in the video frame corresponding to the second time slice, the server or user needs to determine the area that the user has watched in each frame of the 30 panoramic video frames that the user has watched, and display the pixels corresponding to these areas in red, and display other areas in blue. Further, the 30 generated perspective saliency maps are fused to obtain the historical perspective saliency map corresponding to the first time slice; at the same time, determine the areas in each frame that are likely to interest the user, and display the pixels corresponding to these areas in red, and display other areas in blue. Further, the 30 generated video content saliency maps are fused to obtain the historical video content saliency map corresponding to the first time slice.

[0042] Alternatively, when the user is watching the panoramic video frames corresponding to the first time slice, each time a panoramic video frame is displayed to the user, the perspective saliency map and the video content saliency map corresponding to the panoramic video frame are determined in real time according to the above display method. Further, the determined perspective saliency map and the video content saliency map are integrated into the historical perspective saliency map and the historical video content saliency map corresponding to the time slice, thereby realizing the update of the historical perspective saliency map and the historical video content saliency map. Of course, in the actual application process, the method of determining the above two saliency maps can be selected according to actual needs, and the embodiment of the present disclosure does not make specific limitations on this.

[0043] Optionally, in the process of determining the historical perspective saliency map and the historical video content saliency map, the historical perspective information of each historical video frame in the historical time slice can also be determined; the historical perspective saliency map corresponding to each historical perspective information in the corresponding historical video frame and the historical video content saliency map of each historical video frame can be determined.

[0044] Specifically, when a user uses an AR device or a VR device to watch a panoramic video, the server or the user can also obtain historical perspective information. Among them, for a panoramic video frame that has been played, the historical perspective information is information that reflects the user's viewing perspective in the video frame, for example, the field of view (FOV) corresponding to the user when watching the panoramic video frame. It can be understood that the size of the field of view determines the user's field of view. At the same time, based on historical perspective information such as the field of view, the server or the user can at least determine which area of ​​the panoramic picture the user mainly watched when watching the panoramic video frame. For example, when a user watches a panoramic video through an AR device or a VR device, the inertial sensor (Inertial Measurement Unit, IMU) in the device can be used to determine the user's field of view at each moment, wherein the IMU is a sensor for detecting and measuring acceleration and rotational motion. Further, when the model for predicting the user's perspective is deployed on the server, after the user obtains the FOV information of the user at each moment based on the IMU, the information can be fed back to the server, so that the server performs subsequent processing according to the FOV information.

[0045] Furthermore, for each historical video frame, according to the historical perspective information of the current historical video frame, the salient area corresponding to the historical perspective information is determined, and the perspective salient map to be fused is determined based on the salient area; and, based on the salient area segmentation model, the segmentation processing of the current historical video frame is determined to determine the content salient map to be fused of the current historical video frame; based on the perspective salient maps to be fused of each historical video frame, the historical perspective salient map is determined; and, based on the content salient maps to be fused of each historical video frame, the historical video content salient map is determined.

[0046] Among them, the salient area is the area in the panoramic video frame watched by the user at a specific field of view. Since the time slice can correspond to multiple panoramic video frames, after determining the salient area in any panoramic video frame, it can be displayed in red or other highlighted colors, and the non-salient area can be displayed in blue or other dim colors. On this basis, the generated salient map corresponding to the single panoramic video frame is the salient map of the perspective to be fused. Further, multiple salient maps of the perspective to be fused are fused to obtain a historical perspective salient map corresponding to the time slice. The salient area segmentation model can be a pre-trained neural network model, whose input is multiple panoramic video frames. At the same time, the salient area segmentation model can output a salient map of the content to be fused by identifying and dividing the video screen. Similarly, after the model outputs the salient map of the content to be fused corresponding to each panoramic video frame, the multiple salient maps of the content to be fused can be fused to obtain a historical video content salient map corresponding to the time slice.

[0047] Exemplarily, when all panoramic video frames contained in a certain historical time slice are played, on the one hand, the server can obtain 30 viewing angles of 30 frames corresponding to the time slice during the user's viewing process, and determine which area of ​​each panoramic video frame the user mainly watched based on these viewing angles, that is, determine the salient area in each panoramic video frame, thereby generating 30 corresponding perspective salient maps to be fused, and further, after the perspective salient maps to be fused are fused, a historical perspective salient map corresponding to the historical time slice is obtained. On the other hand, the 30 panoramic video frames corresponding to the historical time slice can be input into a pre-trained salient area segmentation model, thereby obtaining 30 corresponding content salient maps to be fused, and further, after the content salient maps to be fused are fused, a historical video content salient map corresponding to the historical time slice is obtained.

[0048] S120, inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of a predicted time slice.

[0049] In this embodiment, after obtaining the historical perspective saliency map and the historical video content saliency map corresponding to the historical time slice, it is also necessary to determine the current perspective saliency map, wherein the current perspective saliency map is the perspective saliency map corresponding to the panoramic video frame currently watched by the user and is used to input into the prediction model. It can be understood that the saliency map can at least reflect which part of the panoramic video frame the user is currently watching.

[0050] Furthermore, the above three saliency maps are input into the pre-trained target perspective prediction model to obtain the predicted video content saliency map of the predicted time slice (i.e., the next time slice of the historical time slice). Among them, the target perspective prediction model can be a model composed of a fully convolutional network (FCN) and a long short-term memory network (LSTM). Those skilled in the art should understand that FCN can classify images at the pixel level, thereby solving semantic level problems. At the same time, FCN can receive input images of any size and generate a prediction for each pixel; LSTM is a time recurrent neural network, which is specially designed to solve the long-standing problems of general recurrent neural networks. The embodiments of the present disclosure will not be repeated here. The following is an explanation of the process of determining the predicted video content saliency map based on the model.

[0051] In this embodiment, for the LSTM that processes the above three images, it can be divided into an encoder and a decoder according to the order of data processing involved in the neural network. On this basis, optionally, the encoder in the target perspective prediction model processes the historical perspective saliency map and the historical video content saliency map to determine the encoding parameters; the decoder in the target perspective prediction model processes the encoding parameters and the current perspective saliency map to obtain the predicted perspective saliency map corresponding to the current time slice.

[0052] The encoding parameters are the parameters determined by the LSTM encoder based on the historical perspective saliency map and the historical video content saliency map. At the same time, the parameters can be passed to the LSTM decoder and used as the initialization parameters of the LSTM decoder. When the LSTM decoder obtains the parameters, it can process the current time saliency map to obtain the predicted perspective saliency map. It can be understood that the predicted perspective saliency map is also a panoramic map. Figure 2 This process is described exemplarily.

[0053] See also Figure 2 , the time slice length is one second, and each time slice corresponds to multiple panoramic video frames. At the same time, after the server or user has determined the ten historical video content saliency maps corresponding to the above time slices and determined the corresponding ten historical perspective saliency maps according to the user FOV trajectory, the historical video content saliency maps and historical perspective saliency maps corresponding to the multiple historical time slices can be sequentially input into the FCN in the target perspective prediction model for processing, and then the processing results are input into the LSTM encoder in the target perspective prediction model to obtain the corresponding encoding parameters. Further, the encoding parameters are passed to the LSTM decoder in the target perspective prediction model, and the encoding parameters are used as the initialization parameters of the LSTM decoder. At this time, the current perspective saliency map can be input into the LSTM decoder for processing, and then the processing results and the video content saliency map corresponding to the first predicted time slice are input into the FCN model together, that is, the predicted perspective saliency map corresponding to the predicted time slice is obtained. It can be understood that through the predicted perspective saliency map, at least it can be determined which area of ​​the panoramic picture the user is most likely to watch when watching the panoramic video corresponding to this predicted time slice.

[0054] Continue to see Figure 2 , the predicted perspective saliency map corresponding to the first predicted time slice is input into the LSTM decoder for processing, and the processing result and the video content saliency map corresponding to the second predicted time slice are input into the FCN model together, that is, the predicted perspective saliency map corresponding to the second predicted time slice is obtained. Those skilled in the art should understand that by analogy in this way, the predicted perspective saliency maps corresponding to the subsequent ten predicted time slices can be obtained, and the embodiments of the present disclosure will not go into details about this.

[0055] pass Figure 2 It can be understood that in the actual process of predicting user perspectives, when only a time slice before the current moment is used as the historical time slice, only the user perspective within a predicted time slice after the current moment can be predicted. When multiple time slices are used as historical time slices, the user perspectives within the corresponding number of predicted time slices can also be predicted.

[0056] In the process of determining the predicted perspective saliency map in this embodiment, not only the areas in the panoramic video that the user has viewed are taken into consideration, but also the areas in the panoramic video that may arouse the user's interest. Predictions are made based on the above-mentioned multiple factors, thereby further preventing the user's subsequent viewing perspective from differing too much from the predicted perspective, thereby enhancing the accuracy of the prediction.

[0057] It should be noted that, in the process of training the target perspective prediction model, a training sample set may be determined first, wherein the training sample set includes multiple training samples, and the training samples include a perspective saliency map to be trained within a time slice to be trained, a video content saliency map to be trained, and a predicted video content saliency map to be trained within a predicted time slice to be trained.

[0058] Furthermore, for each training sample, the perspective saliency map to be trained and the video content saliency map to be trained in the current training sample are input into the perspective prediction model to be trained to obtain the output prediction saliency map; based on the output prediction adjustment map and the predicted video content saliency map to be trained in the current training sample, the loss value is determined, and based on the loss value, the model parameters in the perspective prediction model to be trained are corrected; the convergence of the loss function of the perspective prediction model to be trained is taken as the training goal to obtain the target perspective prediction model.

[0059] In this embodiment, the model parameters in the perspective prediction model to be trained can be corrected based on the loss value. Specifically, after the perspective saliency map to be trained and the video content saliency map to be trained in the current training sample are input into the perspective prediction model to be trained, the corresponding prediction adjustment map can be obtained. At this time, multiple loss values ​​can be determined based on the prediction adjustment map and the corresponding prediction video content saliency map to be trained; further, when the model parameters in the perspective prediction model to be trained are corrected using the loss value, the convergence of the loss function can be used as a training target, such as whether the training error is less than the preset error, or whether the error change tends to be stable, or whether the current number of iterations is equal to the preset number. If the convergence condition is detected, such as the training error of the loss function is less than the preset error, or the error change trend tends to be stable, it indicates that the training of the perspective prediction model to be trained is completed, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not currently reached, other training samples can be further obtained to continue training the perspective prediction model to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the trained perspective prediction model can be used as the target perspective prediction model. That is, at this time, the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map are input into the target perspective prediction model to obtain the predicted video content saliency map of the predicted time slice.

[0060] Optionally, in the process of determining each training sample in the training sample set, at least one historical video to be used can be obtained first; for each historical video to be used, the current historical video to be used is divided according to a preset time slice rule to determine at least one sub-video to be trained of a time slice; a video content saliency map to be trained corresponding to the sub-video to be trained and a user perspective to be used of each sub-video to be trained are determined, and a perspective saliency map to be trained corresponding to the sub-video to be trained is determined; based on the time slice identifier of each sub-video to be trained, the video content saliency map to be trained of the next time slice of the current time slice is used as the predicted video content saliency map to be trained of the current time slice; based on the video content saliency map to be trained, the perspective saliency map to be trained and the predicted video content saliency map to be trained of each sub-video to be trained, at least one training sample is determined.

[0061] Exemplarily, after a user has finished watching a panoramic video, the panoramic video can be used as a historical video to be used. Further, the video can be divided according to a preset time slice rule. For example, when the predicted time length is one second, a ten-second video is divided into 10 sub-videos. These sub-videos are sub-videos to be trained, and each sub-video corresponds to a time slice. Further, according to the method of the embodiment of the present disclosure, a corresponding video content saliency map can be determined for each sub-video to be trained as a video content saliency map to be trained. At the same time, a corresponding perspective saliency map can be determined for each sub-video to be trained according to the perspective of the user to be used as a perspective saliency map to be trained.

[0062] While determining the above data as the input of the perspective prediction model to be trained, it is also necessary to determine the output for training the model. Continuing with the above example, the next time slice adjacent to the time slice can be determined based on the time slice identifier of the sub-video to be trained. Furthermore, according to the method of the embodiment of the present disclosure, the video content saliency map corresponding to the time slice is determined as the predicted video content saliency map to be trained, and these data are associated with the above data as the input of the perspective prediction model to be trained, so as to obtain a group of training samples in the training sample set.

[0063] It can be understood that after the model is trained by selecting the to-be-trained perspective saliency map, the to-be-trained video content saliency map corresponding to a time slice before a specific moment, and the to-be-trained predicted video content saliency map corresponding to a time slice after a specific moment, the model can predict the user perspective within a time slice after the current moment for the panoramic video actually played in the subsequent application process; and when the to-be-trained perspective saliency map, the to-be-trained video content saliency map corresponding to multiple time slices before a specific moment, and the to-be-trained predicted video content saliency map corresponding to a corresponding number of time slices after the specific moment are selected to train the model, the model can predict the user perspective within multiple time slices after the current moment for the panoramic video actually played in the subsequent application process.

[0064] Exemplarily, N time slices can be divided from the historical video to be used according to the pre-set prediction time length, and each time slice can be marked as T1, T2, ..., TN in sequence. Based on this, when predicting the perspective prediction model to be trained, the user's perspective in the T2 period can be predicted based on T1, and the user's perspective in the T3 and T4 period can be predicted based on T1 and T2.

[0065] Specifically, if the model is trained by predicting the user's perspective in the T2 period according to T1, and the model is deployed to the server or user for application, the saliency map corresponding to the slice consistent with the T1 duration can be input into the model for processing, so as to predict the user's perspective in the subsequent slice consistent with the T2 duration; correspondingly, if the model is trained by predicting the user's perspective in the T3 and T4 periods according to T1 and T2, and the model is deployed to the server or user for application, the saliency map corresponding to the slice consistent with the total duration of T1 and T2 can also be input into the model for processing, so as to predict the user's perspective in the subsequent slice consistent with the total duration of T3 and T4. It can be understood that during the training process, the number of time slices used for training can be selected according to actual conditions, and accordingly, the duration of the slice predicted after the model training is completed should also be consistent with the total duration of the time slices used during training, and the embodiments of the present disclosure do not make specific limitations on this.

[0066] S130: Determine a target video within a prediction time slice based on the predicted video content saliency map.

[0067] In this embodiment, when the target perspective prediction model outputs the predicted video content saliency map, the server or client can determine the corresponding target video. The target video can be a video that transmits the pictures of different areas in the panoramic video to the user end at differentiated bit rates and displays it to the user through an AR device or a VR device.

[0068] Optionally, based on the predicted viewing angle saliency map, the predicted viewing angle corresponding to each predicted video frame in the predicted time slice is determined, and the bit rate information of each area in the predicted viewing angle saliency map is adjusted based on the predicted viewing angle, so as to send the target video in the predicted time slice based on the bit rate information.

[0069] The predicted viewing angle is the angle of view that the user is most likely to choose when watching the panoramic video frame contained in the predicted time slice. Bit rate is the bit rate, which refers to the number of bits transmitted by the server per unit time, in bps. Those skilled in the art should understand that based on the bit rate information of a certain area in the panoramic video screen, it can at least be determined whether the relevant data of the area is encoded and transmitted to the user end in low definition, high definition or ultra-high definition.

[0070] Exemplarily, after obtaining the predicted viewing angle saliency map corresponding to the predicted time slice, the area that the user is most likely to view in the subsequent process can be determined by the red area and the blue area in each image. Furthermore, the corresponding viewing angle is determined according to the red area in the predicted viewing angle saliency map, and the picture under this angle is transmitted in high-definition or ultra-high-definition encoding. Correspondingly, the corresponding non-viewing angle is determined according to the blue area in the predicted viewing angle saliency map, and the pictures under these angles are transmitted in low-definition encoding, so that the user end receives the panoramic video under differentiated bit rate transmission. It can be understood that the panoramic video corresponding to the received predicted time slice is the target video.

[0071] It should be noted that in the embodiment of the present disclosure, the bit rate information of each area in the panoramic video to be transmitted can be determined by the server, or the model can be integrated into the client, and the client determines the bit rate information of each area in the panoramic video to be transmitted, and feeds back the determined bit rate information to the server. Finally, the server transmits the panoramic video corresponding to the predicted time slice to the user end according to the received bit rate information. Those skilled in the art should understand that the model can be deployed to the server or the client according to actual needs, and the embodiment of the present disclosure does not make specific limitations here.

[0072] It should also be noted that in actual application, the predicted results may be inconsistent with the user's actual viewing angle. Therefore, a thread can be added in advance to monitor the user's subsequent viewing process. On this basis, when it is monitored that the predicted results are inconsistent with the user's viewing angle, the server can add a fast punch-pull stream to send the picture of the user's actual viewing area to the user end in high-definition or ultra-high-definition encoding, thereby ensuring the user's viewing experience.

[0073] In the actual application process, the solution of the embodiment of the present disclosure can also be based on Figure 3 The flowchart shown is implemented as follows. Figure 2 Provide explanation.

[0074] See also Figure 2, the FOV viewing angle information of the user in the process of watching the historical panoramic video is first determined through the AR device or VR device used by the user, and the information is recorded. Further, the historical viewing angle saliency map is determined according to the FOV viewing angle information, and the historical video content saliency map is combined and input into the pre-trained target viewing angle prediction model, wherein the target viewing angle prediction model is composed of 2D LSTM and FCN. When the model receives the above two saliency maps, the LSTM encoder can first process these saliency maps to determine the encoding parameters. Further, the encoding parameters are passed to the LSTM decoder in the model as initialization parameters, and the viewing angle saliency map of the current moment is processed based on the LSTM decoder to obtain the predicted viewing angle saliency map of the next time slice adjacent to the historical time slice. According to the predicted viewing angle saliency map, the server can determine the bit rate information of each area in the panoramic video, and then according to the bit rate information, the corresponding data is encoded and transmitted to the user end in a differentiated form.

[0075] The technical solution of the disclosed embodiment determines a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map. Furthermore, the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map are input into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice. Finally, based on the predicted video content saliency map, the target video within the predicted time slice is determined, and a machine learning model is used to predict the user's subsequent viewing perspective, so as to provide the user with a corresponding target video based on the prediction results, thereby improving the user's viewing experience. At the same time, the influence of the video content on the user's viewing perspective is taken into account during the prediction process, thereby improving the accuracy of the prediction.

[0076] Embodiment 2

[0077] Figure 4 This is a schematic diagram of the structure of a video processing device provided in the second embodiment of the present disclosure, such as Figure 4 As shown, the device includes: a historical perspective saliency map determination module 210, a predicted video content saliency map determination module 220 and a target video determination module 230.

[0078] The historical perspective saliency map determining module 210 is used to determine the historical perspective saliency map of the historical time slice and the historical video content saliency map corresponding to the historical perspective saliency image; wherein the historical time slice is the previous time slice of the predicted time slice.

[0079] The predicted video content saliency map determination module 220 is used to input the historical view saliency map, the historical video content saliency map and the current view saliency map into a pre-trained target view prediction model to obtain the predicted video content saliency map of the predicted time slice.

[0080] The target video determination module 230 is configured to determine the target video within the prediction time slice based on the predicted video content saliency map.

[0081] On the basis of the above technical solutions, the historical perspective saliency map determining module 210 includes a historical perspective saliency map determining unit and a historical perspective determining unit.

[0082] A historical perspective saliency map determination unit is used to determine the historical perspective saliency map and the historical video content saliency map of the historical time slice when the last video frame in the historical time slice is obtained; or, when the historical video frame in the historical time slice is obtained, update the historical perspective saliency map and the historical video content saliency map corresponding to the historical time slice based on the historical video frame.

[0083] The historical view determining unit is used to determine the historical view information of each historical video frame in the historical time slice.

[0084] Optionally, the historical perspective saliency map determining unit is further used to determine the historical perspective saliency map corresponding to each historical perspective information in the corresponding historical video frame, and the historical video content saliency map of each historical video frame.

[0085] Optionally, the historical perspective saliency map determination unit is also used to determine, for each historical video frame, a salient area corresponding to the historical perspective information according to the historical perspective information of the current historical video frame, and determine the perspective saliency map to be fused based on the salient area; and, based on the salient area segmentation model, determine the segmentation processing of the current historical video frame to determine the content saliency map to be fused of the current historical video frame; determine the historical perspective saliency map based on the perspective saliency maps to be fused of each historical video frame; and, determine the historical video content saliency map based on the content saliency maps to be fused of each historical video frame.

[0086] On the basis of the above-mentioned technical solutions, the predicted video content saliency map determination module 220 includes a coding parameter determination unit and a predicted viewing angle saliency map determination unit.

[0087] The coding parameter determination unit is used to process the historical view saliency map and the historical video content saliency map based on the encoder in the target view prediction model to determine the coding parameters.

[0088] The predicted view saliency map determining unit is used to process the encoding parameters and the current view saliency map based on the decoder in the target view prediction model to obtain the predicted view saliency map corresponding to the current time slice.

[0089] Optionally, the target video determination module 230 is also used to determine the predicted viewing angle corresponding to each predicted video frame in the predicted time slice based on the predicted viewing angle saliency map, and adjust the bit rate information of each area in the predicted viewing angle saliency map based on the predicted viewing angle, so as to send the target video in the predicted time slice based on the bit rate information.

[0090] Based on the above technical solutions, the video processing device also includes a model training module.

[0091] A model training module is used to determine a training sample set, wherein the training sample set includes multiple training samples, and the training samples include a saliency map of the viewing angle to be trained, a saliency map of the video content to be trained, and a saliency map of the predicted video content to be trained in a time slice to be trained; for each training sample, the saliency map of the viewing angle to be trained and the saliency map of the video content to be trained in the current training sample are input into a viewing angle prediction model to be trained to obtain an output predicted saliency map; based on the output predicted adjustment map and the saliency map of the predicted video content to be trained in the current training sample, a loss value is determined, and based on the loss value, the model parameters in the viewing angle prediction model to be trained are corrected; the convergence of the loss function of the viewing angle prediction model to be trained is taken as a training target to obtain the target viewing angle prediction model.

[0092] Optionally, the model training module is also used to obtain at least one historical video to be used; for each historical video to be used, divide the current historical video to be used according to a preset time slice division rule to determine at least one sub-video to be trained of a time slice; determine a video content saliency map to be trained corresponding to the sub-video to be trained, and a user perspective to be used for each sub-video to be trained, and determine a perspective saliency map to be trained corresponding to the sub-video to be trained; based on the time slice identifier of each sub-video to be trained, use the video content saliency map to be trained of the next time slice of the current time slice as the predicted video content saliency map to be trained of the current time slice; determine at least one training sample based on the video content saliency map to be trained, the perspective saliency map to be trained and the predicted video content saliency map to be trained of each sub-video to be trained.

[0093] On the basis of the above-mentioned technical solutions, the historical video content saliency map, the historical perspective saliency map and the predicted video content saliency map are panoramic maps.

[0094] The technical solution provided in this embodiment determines the historical perspective saliency map of the historical time slice, and the historical video content saliency map corresponding to the historical perspective saliency map. Furthermore, the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map are input into the pre-trained target perspective prediction model to obtain the predicted video content saliency map of the predicted time slice. Finally, based on the predicted video content saliency map, the target video in the predicted time slice is determined, and the machine learning model is used to realize the prediction of the user's subsequent viewing perspective, so as to facilitate the provision of the corresponding target video to the user according to the prediction results, thereby improving the user's viewing experience. At the same time, the influence of the video content on the user's viewing perspective is taken into account in the prediction process, thereby improving the accuracy of the prediction.

[0095] The video processing device provided in the embodiments of the present disclosure can execute the video processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0096] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0097] Embodiment 3

[0098] Figure 5 This is a schematic diagram of the structure of an electronic device provided by the third embodiment of the present disclosure. Figure 5 , which shows an electronic device (eg, Figure 5 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0099] like Figure 5As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a pattern processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An edit / output (I / O) interface 305 is also connected to the bus 304.

[0100] Typically, the following devices may be connected to the I / O interface 305: an editing device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0101] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0102] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0103] The electronic device provided by the embodiment of the present disclosure and the video processing method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0104] Embodiment 4

[0105] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the video processing method provided by the above embodiments is implemented.

[0106] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0107] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0110] Determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map; wherein the historical time slice is a previous time slice of the predicted time slice;

[0111] Inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice;

[0112] Based on the predicted video content saliency map, a target video within the predicted time slice is determined.

[0113] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0116] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] According to one or more embodiments of the present disclosure, [Example 1] provides a video processing method, the method comprising:

[0119] Determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map; wherein the historical time slice is a previous time slice of the predicted time slice;

[0120] Inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice;

[0121] Based on the predicted video content saliency map, a target video within the predicted time slice is determined.

[0122] According to one or more embodiments of the present disclosure, [Example 2] provides a video processing method, the method further comprising:

[0123] Optionally, when the last video frame in the historical time slice is obtained, a historical viewpoint saliency map and a historical video content saliency map of the historical time slice are determined; or,

[0124] When the historical video frames in the historical time slice are obtained, the historical viewpoint saliency map and the historical video content saliency map corresponding to the historical time slice are updated based on the historical video frames.

[0125] According to one or more embodiments of the present disclosure, [Example 3] provides a video processing method, the method further comprising:

[0126] Optionally, determining historical viewing angle information of each historical video frame in the historical time slice;

[0127] The historical perspective saliency map corresponding to each historical perspective information in the corresponding historical video frame and the historical video content saliency map of each historical video frame are determined.

[0128] According to one or more embodiments of the present disclosure, [Example 4] provides a video processing method, the method further comprising:

[0129] Optionally, for each historical video frame, according to the historical perspective information of the current historical video frame, a salient area corresponding to the historical perspective information is determined, and a perspective salient map to be fused is determined based on the salient area; and, based on the salient area segmentation model, a segmentation process is determined for the current historical video frame to determine a content salient map to be fused of the current historical video frame;

[0130] Based on the to-be-fused perspective saliency maps of each historical video frame, the historical perspective saliency map is determined; and based on the to-be-fused content saliency maps of each historical video frame, the historical video content saliency map is determined.

[0131] According to one or more embodiments of the present disclosure, [Example 5] provides a video processing method, the method further comprising:

[0132] Optionally, an encoder in the target view prediction model processes the historical view saliency map and the historical video content saliency map to determine encoding parameters;

[0133] The decoder in the target view prediction model processes the encoding parameters and the current view saliency map to obtain a predicted view saliency map corresponding to the current time slice.

[0134] According to one or more embodiments of the present disclosure, [Example 6] provides a video processing method, the method further comprising:

[0135] Optionally, based on the predicted viewing angle saliency map, the predicted viewing angle corresponding to each predicted video frame in the predicted time slice is determined, and the bit rate information of each area in the predicted viewing angle saliency map is adjusted based on the predicted viewing angle, so as to send the target video in the predicted time slice based on the bit rate information.

[0136] According to one or more embodiments of the present disclosure, [Example 7] provides a video processing method, the method further comprising:

[0137] Optionally, a training sample set is determined, wherein the training sample set includes a plurality of training samples, and the training samples include a to-be-trained viewing angle saliency map within a to-be-trained time slice, a to-be-trained video content saliency map, and a to-be-trained predicted video content saliency map within a to-be-trained predicted time slice;

[0138] For each training sample, inputting the to-be-trained viewpoint saliency map and the to-be-trained video content saliency map in the current training sample into the to-be-trained viewpoint prediction model to obtain an output prediction saliency map;

[0139] Determine a loss value based on the output prediction adjustment map and the saliency map of the predicted video content to be trained in the current training sample, and modify the model parameters in the perspective prediction model to be trained based on the loss value;

[0140] The convergence of the loss function of the to-be-trained perspective prediction model is taken as a training goal to obtain the target perspective prediction model.

[0141] According to one or more embodiments of the present disclosure, [Example 8] provides a video processing method, the method further comprising:

[0142] Optionally, obtaining at least one historical video to be used;

[0143] For each historical video to be used, the current historical video to be used is divided according to a preset time slice rule to determine at least one sub-video to be trained of a time slice;

[0144] Determine a saliency map of the video content to be trained corresponding to the sub-video to be trained, and a user perspective to be used for each sub-video to be trained, and determine a saliency map of the perspective to be trained corresponding to the sub-video to be trained;

[0145] According to the time slice identifier of each sub-video to be trained, the content saliency map of the video to be trained of the next time slice of the current time slice is used as the content saliency map of the predicted video to be trained of the current time slice;

[0146] At least one training sample is determined according to the to-be-trained video content saliency map, the to-be-trained viewing angle saliency map, and the to-be-trained predicted video content saliency map of each to-be-trained sub-video.

[0147] According to one or more embodiments of the present disclosure, [Example 9] provides a video processing method, the method further comprising:

[0148] Optionally, the historical video content saliency map, the historical viewing angle saliency map and the predicted video content saliency map are panoramic maps.

[0149] According to one or more embodiments of the present disclosure, [Example 10] provides a video processing device, the device comprising:

[0150] A historical perspective saliency map determination module, used to determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency image; wherein the historical time slice is a previous time slice of the predicted time slice;

[0151] A predicted video content saliency map determination module is used to input the historical view saliency map, the historical video content saliency map and the current view saliency map into a pre-trained target view prediction model to obtain a predicted video content saliency map of the predicted time slice;

[0152] The target video determination module is used to determine the target video within the predicted time slice based on the predicted video content saliency map.

[0153] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0154] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map; wherein the historical time slice is a previous time slice of the predicted time slice, the historical perspective saliency map is determined based on historical perspective information corresponding to each historical video frame in the historical time slice, the historical perspective saliency map is obtained by fusing perspective saliency maps of multiple historical video frames in the historical time slice, and the historical video content saliency map is obtained by fusing video content saliency maps of multiple historical video frames in the historical time slice; Inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice; Based on the predicted video content saliency map, a target video within the predicted time slice is determined.

2. The method according to claim 1, characterized in that The determining of the historical perspective saliency map of the historical time slice and the historical video content saliency map corresponding to the historical perspective saliency map includes: When the last video frame in the historical time slice is acquired, determining the historical viewpoint saliency map and the historical video content saliency map of the historical time slice; or, When the historical video frames in the historical time slice are obtained, the historical viewpoint saliency map and the historical video content saliency map corresponding to the historical time slice are updated based on the historical video frames.

3. The method according to claim 1, characterized in that: The determining of the historical perspective saliency map of the historical time slice and the historical video content saliency map corresponding to the historical perspective saliency map includes: Determining historical viewing angle information of each historical video frame in the historical time slice; The historical perspective saliency map corresponding to each historical perspective information in the corresponding historical video frame and the historical video content saliency map of each historical video frame are determined.

4. The method according to claim 3, characterized in that The determining of the historical perspective saliency map corresponding to each historical perspective information in the corresponding historical video frame and the historical video content saliency map of each historical video frame includes: For each historical video frame, according to the historical perspective information of the current historical video frame, a salient area corresponding to the historical perspective information is determined, and a perspective salient map to be fused is determined based on the salient area; and, based on the salient area segmentation model, a segmentation process is determined for the current historical video frame, and a content salient map to be fused of the current historical video frame is determined; Based on the to-be-fused perspective saliency maps of each historical video frame, the historical perspective saliency map is determined; and based on the to-be-fused content saliency maps of each historical video frame, the historical video content saliency map is determined.

5. The method according to claim 1, characterized in that The step of inputting the historical perspective saliency map, the historical video content saliency map and the current perspective saliency map into a pre-trained target perspective prediction model to obtain a predicted video content saliency map of the predicted time slice comprises: Processing the historical perspective saliency map and the historical video content saliency map based on an encoder in the target perspective prediction model to determine encoding parameters; The decoder in the target view prediction model processes the encoding parameters and the current view saliency map to obtain a predicted view saliency map corresponding to the current time slice.

6. The method according to claim 1, characterized in that The step of determining a target video within the predicted time slice based on the predicted video content saliency map comprises: Based on the predicted viewing angle saliency map, the predicted viewing angle corresponding to each predicted video frame in the predicted time slice is determined, and the bit rate information of each area in the predicted viewing angle saliency map is adjusted based on the predicted viewing angle, so as to send the target video in the predicted time slice based on the bit rate information.

7. The method according to claim 1, characterized in that Also includes: Determine a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include a to-be-trained viewing angle saliency map within a to-be-trained time slice, a to-be-trained video content saliency map, and a to-be-trained predicted video content saliency map within a to-be-trained predicted time slice; For each training sample, the to-be-trained viewpoint saliency map and the to-be-trained video content saliency map in the current training sample are input into the to-be-trained viewpoint prediction model to obtain an output prediction saliency map; Determine a loss value based on the output prediction adjustment map and the saliency map of the predicted video content to be trained in the current training sample, and modify the model parameters in the perspective prediction model to be trained based on the loss value; The convergence of the loss function of the to-be-trained perspective prediction model is taken as a training goal to obtain the target perspective prediction model.

8. The method according to claim 7, characterized in that The step of determining a training sample set comprises: Obtain at least one historical video to be used; For each historical video to be used, the current historical video to be used is divided according to a preset time slice rule to determine at least one sub-video to be trained of a time slice; Determine a saliency map of the video content to be trained corresponding to the sub-video to be trained, and a user perspective to be used for each sub-video to be trained, and determine a saliency map of the perspective to be trained corresponding to the sub-video to be trained; According to the time slice identifier of each sub-video to be trained, the content saliency map of the video to be trained of the next time slice of the current time slice is used as the content saliency map of the predicted video to be trained of the current time slice; At least one training sample is determined according to the to-be-trained video content saliency map, the to-be-trained viewing angle saliency map, and the to-be-trained predicted video content saliency map of each to-be-trained sub-video.

9. The method according to claim 1, characterized in that: The historical video content saliency map, the historical perspective saliency map and the predicted video content saliency map are panoramic maps.

10. A video processing device, characterized in that: include: A historical perspective saliency map determination module, used to determine a historical perspective saliency map of a historical time slice, and a historical video content saliency map corresponding to the historical perspective saliency map; wherein the historical time slice is a previous time slice of the predicted time slice, the historical perspective saliency map is determined based on historical perspective information corresponding to each historical video frame in the historical time slice, the historical perspective saliency map is obtained by fusing perspective saliency maps of multiple historical video frames in the historical time slice, and the historical video content saliency map is obtained by fusing video content saliency maps of multiple historical video frames in the historical time slice; A predicted video content saliency map determination module is used to input the historical view saliency map, the historical video content saliency map and the current view saliency map into a pre-trained target view prediction model to obtain a predicted video content saliency map of the predicted time slice; The target video determination module is used to determine the target video within the predicted time slice based on the predicted video content saliency map.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in any one of claims 1-9.

12. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the video processing method according to any one of claims 1 to 9 when executed by a computer processor.