A view prediction method, apparatus, device and storage medium

Abstract

The embodiment of the present disclosure provides a view prediction method, device and equipment and a storage medium. The method comprises the following steps: determining a current predicted view corresponding to a current to-be-downloaded slice in a panoramic video based on a head movement track of a user watching the panoramic video; if a current head turning action in the head movement track is detected as an invalid action, obtaining a target candidate view set corresponding to the current to-be-downloaded slice; and correcting the current predicted view based on the target candidate view set to determine a target predicted view corresponding to the current to-be-downloaded slice. Through the technical scheme of the embodiment of the present disclosure, the view prediction deviation caused by the user turning the head too much due to non-video factors can be solved, and the accuracy of view prediction is improved.

Description

Technical Field

[0001] This disclosure relates to computer technology, and more particularly to a viewpoint prediction method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, virtual reality (VR) technology has been widely applied. For example, users can wear VR headsets to watch panoramic videos and switch the viewing angle of the panoramic video by turning their heads.

[0003] Currently, VR headsets typically predict the user's future viewing angle, allowing them to request and download only the video stream from that angle, reducing transmission bandwidth. Existing viewpoint prediction methods are all based on a full-view prediction, which means that when the user turns their head excessively due to non-video factors, the predicted viewpoint can be significantly inaccurate, reducing the accuracy of the viewpoint prediction. Summary of the Invention

[0004] This disclosure provides a viewpoint prediction method, apparatus, device, and storage medium to address viewpoint prediction errors caused by excessive head turning due to non-video factors, thereby improving the accuracy of viewpoint prediction.

[0005] In a first aspect, embodiments of this disclosure provide a viewpoint prediction method, including:

[0006] Based on the head movement trajectory of the user watching the panoramic video, the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video is determined.

[0007] If the current head-turning action in the head motion trajectory is detected as an invalid action, then the target candidate viewpoint set corresponding to the current slice to be downloaded is obtained;

[0008] Based on the set of target candidate perspectives, the current predicted perspective is corrected to determine the target predicted perspective corresponding to the current slice to be downloaded.

[0009] Secondly, embodiments of this disclosure also provide a view prediction device, comprising:

[0010] The current prediction viewpoint determination module is used to determine the current prediction viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the head movement trajectory of the user watching the panoramic video.

[0011] The target candidate view set acquisition module is used to acquire the target candidate view set corresponding to the current slice to be downloaded if the current head turning action in the head movement trajectory is detected as an invalid action.

[0012] The viewpoint correction processing module is used to correct the current predicted viewpoint based on the target candidate viewpoint set, and determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the view prediction method as described in any embodiment of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the view prediction method as described in any of the embodiments of this disclosure.

[0018] In this embodiment, the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video is determined based on the user's head movement trajectory while watching the video, and the current head-turning action in the head movement trajectory is detected as invalid. If the current head-turning action is invalid, it indicates that the user is turning their head significantly due to non-video factors. In this case, the current predicted viewpoint can be corrected based on the set of target candidate viewpoints corresponding to the current slice to be downloaded, thereby obtaining a more accurate target predicted viewpoint for the current slice to be downloaded. This avoids a large deviation in the predicted viewpoint caused by the user turning their head too much due to non-video factors, thus improving the accuracy of viewpoint prediction. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a schematic flowchart of a view prediction method provided in an embodiment of this disclosure;

[0021] Figure 2 This is an example of a perspective prediction process involved in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram of another perspective prediction method provided in the embodiments of this disclosure;

[0023] Figure 4 This is a schematic flowchart of another view prediction method provided in the embodiments of this disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of a view prediction device provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] Figure 1This is a flowchart illustrating a viewpoint prediction method provided in an embodiment of this disclosure. This embodiment is applicable to predicting the viewing angle of a slice to be downloaded in a panoramic video, and is particularly suitable for scenarios involving viewpoint prediction of panoramic videos played on VR headsets. The method can be executed by a viewpoint prediction device, which can be implemented in software and / or hardware, optionally through an electronic device, such as a VR headset.

[0033] like Figure 1 As shown, the viewpoint prediction method specifically includes the following steps:

[0034] S110. Based on the head movement trajectory of the user watching the panoramic video, determine the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video.

[0035] The panoramic video can be, but is not limited to, VR panoramic video. Panoramic videos can be downloaded and played in video slices; that is, a panoramic video can include multiple video slices. The download and playback of the panoramic video is completed by downloading and playing all video slices sequentially. The currently downloaded slice refers to the video slice in the panoramic video that needs to be downloaded at the moment. To reduce the risk of playback stuttering and ensure smooth viewing, the panoramic video playback device will cache some downloaded but not yet played video slices. In other words, there is a certain length of cached video between the currently playing video slice and the currently downloaded slice. For example, Figure 2 An example of a viewpoint prediction process is given. Figure 2 As shown, the playback device is playing video slice 2 from the panoramic video. The slice to be downloaded is video slice M. The video length between the current viewing position in video slice 2 and video slice M is the video buffer length of the playback device.

[0036] The head motion trajectory refers to the change in a user's head position from the start of viewing the panoramic video to the current moment. For example, the head motion trajectory can consist of multiple head positions sampled between the start of viewing the panoramic video and the current moment. The head motion trajectory can be dynamically updated as viewing time progresses. For example, the head motion trajectory can be: [P0, P1, P2, ..., Pt]. Here, P0 refers to the head position at the start of viewing the panoramic video, and Pt refers to the head position at the current moment. It should be noted that the head position corresponds to the viewing angle, and the user can switch the viewing angle by rotating their head. For example, each head position can correspond to a viewing angle, or a certain range of head positions can correspond to a viewing angle; this correspondence can be set based on business requirements. The head motion trajectory can be used to characterize the change in the viewing angle of a panoramic video.

[0037] Specifically, while watching panoramic videos, existing viewpoint prediction methods can be used to predict the current predicted viewpoint corresponding to the current slice to be downloaded, based on the user's head movement trajectory. For example, the viewpoint fluctuation information of the user during the historical viewing time can be determined based on the user's head movement trajectory, and viewpoint prediction can be performed based on the viewpoint fluctuation information and the current viewing viewpoint to obtain the current predicted viewpoint corresponding to the current slice to be downloaded. For example, if the viewpoint fluctuation variance is less than or equal to a preset variance threshold, it indicates that the user's head turning action is relatively stable, and a conservative viewpoint prediction method can be used for prediction. If the viewpoint fluctuation variance is greater than the preset variance threshold, it indicates that the user prefers to turn the viewpoint, and an aggressive viewpoint prediction method can be used for prediction. This can initially avoid the situation where the predicted current viewpoint deviates too much from the current viewing viewpoint, initially ensuring the accuracy of the viewpoint prediction results, and can also further improve the subsequent viewpoint correction effect.

[0038] S120. If the current head-turning action in the head motion trajectory is detected as an invalid action, then obtain the target candidate view set corresponding to the current slice to be downloaded.

[0039] The current head-turning action can refer to the user's head-turning action occurring at the current moment, that is, the action of the user turning from the previous head position to the current head position. The current head position can refer to the user's head position at the current moment. The previous head position can refer to the user's head position at the previous sampling moment. An invalid action can refer to an action of turning the head significantly due to non-video factors. Each slice in the panoramic video corresponds to a set of candidate viewpoints. The target candidate viewpoint set can refer to the set of candidate viewpoints corresponding to the current slice to be downloaded. The target candidate viewpoint set can consist of one or more target candidate viewpoints. The target candidate viewpoint can refer to the viewing angle that the user might be interested in when watching the current slice to be downloaded. The target candidate viewpoint can be determined based on the slice content characteristics corresponding to the current slice to be downloaded and / or the user's actual viewing angle.

[0040] Specifically, based on head movement trajectories, the system can detect whether the user's current head-turning action is invalid. For example, it can use viewpoint tracking and image recognition to determine whether the current head-turning action is an accidental action caused by non-video factors. If the current head-turning action is detected as invalid, the playback device can determine the set of target candidate viewpoints corresponding to the current slice to be downloaded in real time based on the slice content features, or it can directly obtain the set of target candidate viewpoints corresponding to the current slice to be downloaded from the server, thereby further improving the efficiency and accuracy of viewpoint prediction.

[0041] It should be noted that if the current head-turning action is detected as a valid action, it indicates that there is no significant deviation in the current predicted viewpoint. In this case, the server can be directly requested to download the current slice to be downloaded under the current predicted viewpoint, thereby ensuring the accuracy of the viewpoint prediction.

[0042] S130. Based on the set of candidate viewpoints, the current predicted viewpoint is corrected to determine the target predicted viewpoint corresponding to the slice to be downloaded.

[0043] Specifically, the target candidate viewpoint closest to the current predicted viewpoint in the set of candidate viewpoints can be used as the target predicted viewpoint for the current slice to be downloaded. This allows for correction of the current predicted viewpoint using the set of candidate viewpoints, preventing significant shifts in the predicted viewpoint caused by excessive head turning due to non-video factors, thus improving the accuracy of viewpoint prediction. After determining the target predicted viewpoint, a request can be made to the server to download the current slice under that viewpoint, ensuring accurate slice downloading and further saving transmission bandwidth.

[0044] For example, S130 may include: determining the view difference between each target candidate view in the target candidate view set and the current predicted view; and determining the target candidate view with the smallest view difference as the target predicted view corresponding to the current slice to be downloaded.

[0045] Specifically, each candidate viewpoint in the set of candidate viewpoints can be subtracted from the current predicted viewpoint, and the absolute value of the subtraction is determined as the corresponding viewpoint difference. All viewpoint differences are compared, and the candidate viewpoint with the smallest viewpoint difference is determined as the target predicted viewpoint. This allows for accurate viewpoint correction and ensures the accuracy of viewpoint prediction.

[0046] The technical solution of this disclosure determines the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the user's head movement trajectory while watching the video, and detects whether the current head-turning action in the head movement trajectory is an invalid action. If the current head-turning action is invalid, it indicates that the user is turning their head significantly due to non-video factors. In this case, the current predicted viewpoint can be corrected based on the set of target candidate viewpoints corresponding to the current slice to be downloaded, thereby obtaining a more accurate target predicted viewpoint corresponding to the current slice to be downloaded. This avoids a large deviation in the predicted viewpoint caused by the user turning their head too much due to non-video factors, and improves the accuracy of viewpoint prediction.

[0047] Based on the above technical solution, the "obtaining the target alternative viewpoint set corresponding to the current slice to be downloaded" in S120 may include: obtaining the panoramic video header file corresponding to the panoramic video, wherein the panoramic video header file includes: the alternative viewpoint set corresponding to each slice of the panoramic video, the alternative viewpoint set being determined based on the slice content features corresponding to each slice and / or the user's actual viewing angle; and obtaining the target alternative viewpoint set corresponding to the current slice to be downloaded based on the panoramic video header file.

[0048] Specifically, such as Figure 2 As shown, the server can pre-determine the set of candidate viewpoints for each slice based on the slice content features and / or the user's actual viewing angle. For example, it can determine the viewing direction of the area the user might be interested in based solely on the slice content features, thus obtaining the set of candidate viewpoints for each slice; alternatively, it can collect the actual viewing angles of each slice when all users watch the panoramic video, rank them by popularity based on the actual viewing angles, and use the most popular actual viewing angles as candidate viewpoints for that slice, i.e., use the actual viewing angles that users watch most often as candidate viewpoints for that slice, thus obtaining the set of candidate viewpoints for that slice; it can also determine the set of candidate viewpoints for each slice based on both slice content features and the user's actual viewing angle, in order to further improve the accuracy of viewpoint prediction. After determining the set of candidate viewpoints for each slice, the server can write all the candidate viewpoint sets into the panoramic video header file and send them to the playback device simultaneously with the panoramic video. The playback device can obtain the target set of candidate viewpoints for the currently downloaded slice more quickly based on the set of candidate viewpoints for each slice in the panoramic video header file, thus further improving the efficiency of viewpoint prediction.

[0049] For example, determining the set of candidate viewpoints based on the slice content features corresponding to each slice and the user's actual viewing perspective may include: for each slice, determining all viewpoints of interest corresponding to the slice based on the slice content features corresponding to the slice; determining the viewing popularity of each viewpoint of interest based on the user's actual viewing perspective corresponding to the slice; and selecting each viewpoint of interest with a viewing popularity greater than or equal to a preset popularity threshold as a candidate viewpoint to obtain the set of candidate viewpoints corresponding to the slice.

[0050] Specifically, for each slice, the content characteristics of the slice under different viewing angles can be analyzed to determine all the viewing angles of interest to users. For example, if the slice content is a soccer ball kicking into a goal, then the viewing angle that does not obstruct the soccer ball can be determined as the viewing angle of interest. Statistical analysis of the actual viewing angles of users corresponding to the slice is performed to determine the number of users who use the viewing angle of interest as their actual viewing angle, i.e., the viewing popularity of the viewing angle of interest. Each viewing angle of interest with a viewing popularity greater than or equal to a preset popularity threshold is selected as a candidate viewing angle, obtaining the candidate viewing angle set corresponding to the slice. Thus, by simultaneously utilizing the slice content characteristics and the actual viewing angles of users, the candidate viewing angle set corresponding to each slice can be determined more accurately, further improving the accuracy of viewing angle prediction.

[0051] Figure 3 This is a flowchart illustrating another perspective prediction method provided in this embodiment. Based on the above-described embodiments, this embodiment further optimizes the step of "determining the current predicted perspective corresponding to the current slice to be downloaded in the panoramic video based on the user's head movement trajectory while watching the panoramic video." Explanations of terms identical or corresponding to those in the above-described embodiments are not repeated here.

[0052] like Figure 3 As shown, the viewpoint prediction method specifically includes the following steps:

[0053] S310. Based on the head position in the head movement trajectory of the user watching the panoramic video, determine the standard deviation of the viewpoint and the gradient of the viewpoint change between the current head position and the previous head position.

[0054] The standard deviation of the viewing angle can be used to characterize the change in viewing angle during a user's viewing of a panoramic video. A larger standard deviation indicates that the user prefers to rotate the viewing angle. The gradient of the viewing angle change can refer to the rate of change of the viewing angle during the current head-turning action. Head position corresponds to the viewing angle. For example, each head position can correspond to a viewing angle, or a range of head position changes can correspond to a viewing angle.

[0055] Specifically, the viewing angle corresponding to each head position in the user's head movement trajectory while watching the panoramic video can be obtained, and the standard deviation of all viewing angles during the user's viewing process can be calculated to obtain the viewing angle standard deviation. Based on the current viewing angle corresponding to the current head position, the previous viewing angle corresponding to the previous head position, and the time interval between the current head position and the previous head position, the viewing angle change gradient of the user's current head turning action can be determined.

[0056] S320. Determine the target view interval based on the current video buffer length and the view change gradient.

[0057] The current video buffer length can refer to the duration of the video currently buffered by the playback device. The target viewpoint interval can be used to characterize the degree of viewpoint deviation between the current moment and the future playback moment of the currently downloaded segment.

[0058] Specifically, the current video buffer length and the gradient of the viewpoint change can be multiplied together, and the result of the multiplication can be determined as the target viewpoint interval.

[0059] S330. Based on the current viewing angle, target viewing angle interval, and viewing angle standard deviation, determine the current predicted viewing angle corresponding to the current slice to be downloaded.

[0060] Specifically, based on the target viewpoint interval and viewpoint standard deviation, it is possible to detect whether the viewpoint deviation between the current moment and the future playback moment of the current slice to be downloaded is too large. Based on the detection results, the current viewing viewpoint is processed to obtain the current predicted viewpoint corresponding to the current slice to be downloaded. This can avoid the situation where the viewpoint prediction result is large due to the large length of the current video buffer, further ensuring the accuracy of the predicted viewpoint, and thus further improving the subsequent viewpoint correction effect.

[0061] For example, S330 may include: determining a viewpoint interval threshold based on the viewpoint standard deviation; if the target viewpoint interval is greater than the viewpoint interval threshold, determining the current predicted viewpoint corresponding to the current slice to be downloaded based on the current viewing viewpoint and the viewpoint interval threshold; if the target viewpoint interval is less than or equal to the viewpoint interval threshold, determining the current predicted viewpoint corresponding to the current slice to be downloaded based on the current viewing viewpoint and the target viewpoint interval.

[0062] Specifically, based on the characteristics of the Gaussian distribution, twice the standard deviation of the viewing angle can be determined as the viewing angle interval threshold to ensure that most prediction scenarios can be covered. When the target viewing angle interval is less than or equal to the viewing angle interval threshold, the current viewing angle and the target viewing angle interval are added together, and the result is determined as the current predicted viewing angle corresponding to the current slice to be downloaded. When the target viewing angle interval is greater than the viewing angle interval threshold, the current viewing angle and the viewing angle interval threshold are added together, and the result is determined as the current predicted viewing angle corresponding to the current slice to be downloaded. This limits the prediction results and avoids excessive deviation.

[0063] S340. If the current head-turning action in the head motion trajectory is detected as an invalid action, then obtain the target candidate view set corresponding to the current slice to be downloaded.

[0064] S350. Based on the set of candidate viewpoints, the current predicted viewpoint is corrected to determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0065] The technical solution of this disclosure determines the target viewpoint interval based on the current video buffer length and the viewpoint change gradient between the current head position and the previous head position. Based on the current viewing angle, the target viewpoint interval, and the viewpoint standard deviation, the current predicted viewpoint corresponding to the current slice to be downloaded is determined. This can avoid the situation where the viewpoint prediction result is greatly deviated due to the large current video buffer length, further ensuring the accuracy of the current predicted viewpoint, and further improving the subsequent viewpoint correction effect.

[0066] Figure 4 This is a flowchart illustrating another perspective prediction method provided in this disclosure. Based on the above-described embodiments, this disclosure provides a detailed description of the detection method for whether the current head-turning action is an invalid action. Explanations of terms that are the same as or corresponding to those in the above-described embodiments are not repeated here.

[0067] like Figure 4 As shown, the viewpoint prediction method specifically includes the following steps:

[0068] S410. Based on the head movement trajectory of the user watching the panoramic video, determine the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video.

[0069] S420. Determine the head turning amplitude corresponding to the current head turning action based on the current head position and the previous head position in the head movement trajectory.

[0070] Specifically, the user's current head position can be subtracted from the previous head position to obtain the head turning range corresponding to the current head turning action.

[0071] S430. If the head turning amplitude is greater than or equal to the preset amplitude threshold, then determine the number of significant changes in video content corresponding to the current head turning action.

[0072] The preset amplitude threshold can refer to the minimum head turning amplitude required for viewpoint correction. Specifically, when the head turning amplitude corresponding to the user's current head turning action is greater than or equal to the preset amplitude threshold, it indicates that the user's head turning amplitude is too large and the head movement trajectory has changed drastically. At this time, the degree of change in video content during the current head turning action can be identified to determine the number of significant changes in video content.

[0073] It should be noted that if the head turning amplitude is less than the preset amplitude threshold, it indicates that the current head turning amplitude is not large. The current predicted viewpoint predicted based on the head movement trajectory has been corrected to a certain extent. At this time, the current head turning action can be determined as a valid action, and there is no need to use the alternative viewpoint set for correction, thereby further ensuring the viewpoint correction effect.

[0074] For example, "determining the number of significant changes in video content corresponding to the current head-turning action" in S430 may include: acquiring all target video frames corresponding to the current head-turning action in the panoramic video; determining the image structure similarity between every two adjacent target video frames; and determining the number of image structure similarities greater than or equal to a preset similarity threshold as the number of significant changes in video content.

[0075] Image structural similarity measures the degree of similarity between the content of two images. A target video frame refers to the video frame viewed during the current head-turning action. Different target video frames correspond to different viewing angles.

[0076] Specifically, all video frames that the user will see during the current head turn can be used as target video frames. Based on the video content of each pair of adjacent target video frames, the image structure similarity between each pair of adjacent target video frames is determined, and based on the image structure similarity, it is determined whether there are significant changes in the video content between each pair of adjacent target video frames. For example, if the image structure similarity between two adjacent target video frames is greater than or equal to a preset similarity threshold, the number of significant changes in video content is incremented by 1, thus determining the number of image structure similarities greater than or equal to the preset similarity threshold as the number of significant changes in video content.

[0077] S440. If the number of significant changes in video content is greater than or equal to a preset threshold, then the current head-turning action is determined to be an invalid action.

[0078] The preset threshold number refers to the minimum number of significant changes in the video content that require perspective correction. Specifically, if the number of significant changes in the video content is greater than or equal to the preset threshold number, the current head-turning action can be determined as an invalid action due to accidental head turning caused by non-video factors. If the number of significant changes in the video content is less than the preset threshold number, the current head-turning action can be determined as a valid action, eliminating the need for correction processing using the alternative perspective set, thereby further ensuring the effectiveness of perspective correction.

[0079] S450: Obtain the set of target candidate viewpoints corresponding to the current slice to be downloaded.

[0080] S460. Based on the set of target candidate viewpoints, the current predicted viewpoint is corrected to determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0081] The technical solution of this disclosure determines the head turning amplitude corresponding to the current head turning action based on the current head position and the previous head position in the head movement trajectory; if the head turning amplitude is greater than or equal to a preset amplitude threshold, the number of significant changes in video content corresponding to the current head turning action is determined; if the number of significant changes in video content is greater than or equal to a preset number threshold, the current head turning action can be determined as an invalid action due to accidental head turning caused by non-video factors, thereby further ensuring the viewpoint correction effect.

[0082] Figure 5 This is a schematic diagram of the structure of a view prediction device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the device specifically includes: a current prediction view determination module 510, a target alternative view set acquisition module 520, and a view correction processing module 530.

[0083] The current predicted viewpoint determination module 510 is used to determine the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the head movement trajectory of the user watching the panoramic video; the target candidate viewpoint set acquisition module 520 is used to acquire the target candidate viewpoint set corresponding to the current slice to be downloaded if the current head turning action in the head movement trajectory is detected as an invalid action; and the viewpoint correction processing module 530 is used to perform correction processing on the current predicted viewpoint based on the target candidate viewpoint set to determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0084] The technical solution provided in this disclosure determines the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the user's head movement trajectory while watching the video, and detects whether the current head-turning action in the head movement trajectory is an invalid action. If the current head-turning action is invalid, it indicates that the user is turning their head significantly due to non-video factors. In this case, the current predicted viewpoint can be corrected based on the set of target candidate viewpoints corresponding to the current slice to be downloaded, thereby obtaining a more accurate target predicted viewpoint corresponding to the current slice to be downloaded. This avoids a large deviation in the predicted viewpoint caused by the user turning their head too much due to non-video factors, and improves the accuracy of viewpoint prediction.

[0085] Based on the above technical solution, the current prediction perspective determination module 510 includes:

[0086] The viewing angle change gradient determination unit is used to determine the viewing angle standard deviation and the viewing angle change gradient between the current head position and the previous head position based on the head position in the head movement trajectory of the user watching the panoramic video, wherein the head position corresponds to the viewing angle.

[0087] The target viewpoint interval determination unit is used to determine the target viewpoint interval based on the current video buffer length and the viewpoint change gradient.

[0088] The current prediction viewpoint determination unit is used to determine the current prediction viewpoint corresponding to the current slice to be downloaded based on the current viewing viewpoint, the target viewpoint interval, and the viewpoint standard deviation.

[0089] Based on the above technical solutions, the current prediction perspective determination unit is specifically used for:

[0090] A viewing angle interval threshold is determined based on the viewing angle standard deviation; if the target viewing angle interval is greater than the viewing angle interval threshold, the current predicted viewing angle corresponding to the current slice to be downloaded is determined based on the current viewing angle and the viewing angle interval threshold; if the target viewing angle interval is less than or equal to the viewing angle interval threshold, the current predicted viewing angle corresponding to the current slice to be downloaded is determined based on the current viewing angle and the target viewing angle interval.

[0091] Based on the above technical solutions, the device further includes: a head rotation motion detection module, comprising:

[0092] The head turning amplitude determination unit is used to determine the head turning amplitude corresponding to the current head turning action based on the current head position and the previous head position in the head movement trajectory.

[0093] The change count determination unit is used to determine the number of significant changes in the video content corresponding to the current head turning action if the head turning amplitude is greater than or equal to a preset amplitude threshold.

[0094] The invalid action determination unit is used to determine the current head-turning action as invalid if the number of significant changes in the video content is greater than or equal to a preset threshold number.

[0095] Based on the above technical solutions, the change number determination unit is specifically used for:

[0096] Obtain all target video frames corresponding to the current head-turning action in the panoramic video; determine the image structure similarity between each pair of adjacent target video frames; and determine the number of image structure similarities greater than or equal to a preset similarity threshold as the number of significant changes in video content.

[0097] Based on the above technical solutions, the target candidate view set acquisition module 520 is specifically used for:

[0098] Obtain the panoramic video header file corresponding to the panoramic video, wherein the panoramic video header file includes: a set of alternative viewpoints corresponding to each slice of the panoramic video, the set of alternative viewpoints being determined based on the slice content features corresponding to each slice and / or the user's actual viewing angle; based on the panoramic video header file, obtain the set of target alternative viewpoints corresponding to the slice to be downloaded.

[0099] Based on the above technical solutions, the device also includes:

[0100] The alternative viewpoint set determination module is used to determine all viewpoints of interest corresponding to each slice based on the slice content characteristics; determine the viewing popularity of each viewpoint of interest based on the user's actual viewing perspective corresponding to the slice; and select each viewpoint of interest with a viewing popularity greater than or equal to a preset popularity threshold as an alternative viewpoint to obtain the alternative viewpoint set corresponding to the slice.

[0101] Based on the above technical solutions, the viewing angle correction processing module 530 is specifically used for:

[0102] Determine the view difference between each target candidate view in the target candidate view set and the current predicted view; determine the target candidate view with the smallest view difference as the target predicted view corresponding to the current slice to be downloaded.

[0103] The view prediction device provided in this disclosure can execute the view prediction method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the view prediction method.

[0104] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0105] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0106] like Figure 6 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0107] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0108] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0109] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0110] The electronic device provided in this embodiment and the view prediction method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0111] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the viewpoint prediction method provided in the above embodiments.

[0112] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0115] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0116] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the head movement trajectory of the user watching the panoramic video; if the current head-turning action in the head movement trajectory is detected as an invalid action, obtain a set of target candidate viewpoints corresponding to the current slice to be downloaded; and, based on the set of target candidate viewpoints, perform correction processing on the current predicted viewpoint to determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0117] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0120] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] According to one or more embodiments of this disclosure, [Example 1] provides a viewpoint prediction method, including:

[0123] Based on the head movement trajectory of the user watching the panoramic video, the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video is determined.

[0124] If the current head-turning action in the head motion trajectory is detected as an invalid action, then the target candidate viewpoint set corresponding to the current slice to be downloaded is obtained;

[0125] Based on the set of target candidate perspectives, the current predicted perspective is corrected to determine the target predicted perspective corresponding to the current slice to be downloaded.

[0126] According to one or more embodiments of this disclosure, [Example 2] provides a viewpoint prediction method, which further includes:

[0127] Optionally, determining the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the user's head movement trajectory while watching the panoramic video includes:

[0128] Based on the head position in the head movement trajectory of the user watching the panoramic video, the standard deviation of the viewpoint and the gradient of the viewpoint change between the current head position and the previous head position are determined, wherein the head position corresponds to the viewing viewpoint.

[0129] The target view interval is determined based on the current video buffer length and the view change gradient.

[0130] Based on the current viewing angle, the target viewing angle interval, and the viewing angle standard deviation, the current predicted viewing angle corresponding to the current slice to be downloaded is determined.

[0131] According to one or more embodiments of this disclosure, [Example 3] provides a viewpoint prediction method, which further includes:

[0132] Optionally, determining the current predicted viewpoint corresponding to the current slice to be downloaded based on the current viewing angle, the target viewpoint interval, and the viewpoint standard deviation includes:

[0133] The viewing angle interval threshold is determined based on the viewing angle standard deviation.

[0134] If the target viewpoint interval is greater than the viewpoint interval threshold, then the current predicted viewpoint corresponding to the current slice to be downloaded is determined based on the current viewing viewpoint and the viewpoint interval threshold.

[0135] If the target view interval is less than or equal to the view interval threshold, then the current predicted view corresponding to the current slice to be downloaded is determined based on the current viewing view and the target view interval.

[0136] According to one or more embodiments of this disclosure, [Example 4] provides a view prediction method, which further includes:

[0137] Optionally, detecting that the current head-turning action in the head movement trajectory is an invalid action includes:

[0138] Based on the current head position and the previous head position in the head movement trajectory, determine the head turning amplitude corresponding to the current head turning action;

[0139] If the head-turning amplitude is greater than or equal to a preset amplitude threshold, then the number of significant changes in video content corresponding to the current head-turning action is determined.

[0140] If the number of significant changes in the video content is greater than or equal to a preset threshold, then the current head-turning action is determined to be an invalid action.

[0141] According to one or more embodiments of this disclosure, Example 5 provides a view prediction method, which further includes:

[0142] Optionally, determining the number of significant changes in video content corresponding to the current head-turning action includes:

[0143] Obtain all target video frames corresponding to the current head-turning action in the panoramic video;

[0144] Determine the image structural similarity between every two adjacent target video frames;

[0145] The number of image structural similarities that are greater than or equal to a preset similarity threshold is determined as the number of significant changes in video content.

[0146] According to one or more embodiments of this disclosure, [Example Six] provides a view prediction method, further comprising:

[0147] Optionally, obtaining the target candidate view set corresponding to the current slice to be downloaded includes:

[0148] Obtain the panoramic video header file corresponding to the panoramic video, wherein the panoramic video header file includes: a set of alternative viewpoints corresponding to each slice of the panoramic video, the set of alternative viewpoints being determined based on the slice content features corresponding to each slice and / or the user's actual viewing angle;

[0149] Based on the panoramic video header file, a set of target candidate viewpoints corresponding to the current slice to be downloaded is obtained.

[0150] According to one or more embodiments of this disclosure, [Example Seven] provides a view prediction method, further comprising:

[0151] Optionally, a set of alternative viewing angles is determined based on the slice content features corresponding to each slice and the user's actual viewing angle, including:

[0152] For each slice, all viewpoints of interest corresponding to that slice are determined based on the slice content features.

[0153] Based on the user's actual viewing perspective corresponding to the slice, determine the viewing popularity of each of the aforementioned perspectives of interest;

[0154] Each viewpoint of interest with a viewing popularity greater than or equal to a preset popularity threshold is selected as a candidate viewpoint, thus obtaining a set of candidate viewpoints corresponding to the slice.

[0155] According to one or more embodiments of this disclosure, [Example Eight] provides a view prediction method, further comprising:

[0156] Optionally, the step of correcting the current predicted view based on the set of target candidate views to determine the target predicted view corresponding to the current slice to be downloaded includes:

[0157] Determine the view difference between each target candidate view in the target candidate view set and the current predicted view;

[0158] The candidate viewpoint with the smallest viewpoint difference is determined as the target prediction viewpoint corresponding to the current slice to be downloaded.

[0159] According to one or more embodiments of this disclosure, [Example Nine] provides a view prediction device, including:

[0160] The current prediction viewpoint determination module is used to determine the current prediction viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the head movement trajectory of the user watching the panoramic video.

[0161] The target candidate view set acquisition module is used to acquire the target candidate view set corresponding to the current slice to be downloaded if the current head turning action in the head movement trajectory is detected as an invalid action.

[0162] The viewpoint correction processing module is used to correct the current predicted viewpoint based on the target candidate viewpoint set, and determine the target predicted viewpoint corresponding to the current slice to be downloaded.

[0163] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0164] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0165] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A viewpoint prediction method, characterized in that, include: Based on the head movement trajectory of the user watching the panoramic video, the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video is determined. If the current head-turning action in the head motion trajectory is detected as an invalid action, then the target candidate viewpoint set corresponding to the current slice to be downloaded is obtained; Determine the view difference between each target candidate view in the target candidate view set and the current predicted view, and determine the target candidate view with the smallest view difference as the target predicted view corresponding to the current slice to be downloaded.

2. The viewpoint prediction method according to claim 1, characterized in that, The process of determining the current predicted viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the user's head movement trajectory while watching the panoramic video includes: Based on the head position in the head movement trajectory of the user watching the panoramic video, the standard deviation of the viewpoint and the gradient of the viewpoint change between the current head position and the previous head position are determined, wherein the head position corresponds to the viewing viewpoint. The target view interval is determined based on the current video buffer length and the view change gradient. Based on the current viewing angle, the target viewing angle interval, and the viewing angle standard deviation, the current predicted viewing angle corresponding to the current slice to be downloaded is determined.

3. The viewpoint prediction method according to claim 2, characterized in that, The step of determining the current predicted viewpoint corresponding to the current slice to be downloaded based on the current viewing angle, the target viewpoint interval, and the viewpoint standard deviation includes: The viewing angle interval threshold is determined based on the viewing angle standard deviation. If the target viewpoint interval is greater than the viewpoint interval threshold, then the current predicted viewpoint corresponding to the current slice to be downloaded is determined based on the current viewing viewpoint and the viewpoint interval threshold. If the target view interval is less than or equal to the view interval threshold, then the current predicted view corresponding to the current slice to be downloaded is determined based on the current viewing view and the target view interval.

4. The viewpoint prediction method according to claim 1, characterized in that, Detecting the current head-turning motion in the head movement trajectory as an invalid motion includes: Based on the current head position and the previous head position in the head movement trajectory, determine the head turning amplitude corresponding to the current head turning action; If the head-turning amplitude is greater than or equal to a preset amplitude threshold, then the number of significant changes in video content corresponding to the current head-turning action is determined. If the number of significant changes in the video content is greater than or equal to a preset threshold, then the current head-turning action is determined to be an invalid action.

5. The viewpoint prediction method according to claim 4, characterized in that, The determination of the number of significant changes in video content corresponding to the current head-turning action includes: Obtain all target video frames corresponding to the current head-turning action in the panoramic video; Determine the image structural similarity between every two adjacent target video frames; The number of image structural similarities that are greater than or equal to a preset similarity threshold is determined as the number of significant changes in video content.

6. The viewpoint prediction method according to claim 1, characterized in that, The step of obtaining the target candidate view set corresponding to the current slice to be downloaded includes: Obtain the panoramic video header file corresponding to the panoramic video, wherein the panoramic video header file includes: a set of alternative viewpoints corresponding to each slice of the panoramic video, the set of alternative viewpoints being determined based on the slice content features corresponding to each slice and / or the user's actual viewing angle; Based on the panoramic video header file, a set of target candidate viewpoints corresponding to the current slice to be downloaded is obtained.

7. The viewpoint prediction method according to claim 6, characterized in that, A set of candidate viewpoints is determined based on the content features of each slice and the user's actual viewing perspective, including: For each slice, all viewpoints of interest corresponding to that slice are determined based on the slice content features. Based on the user's actual viewing perspective corresponding to the slice, determine the viewing popularity of each of the aforementioned perspectives of interest; Each viewpoint of interest with a viewing popularity greater than or equal to a preset popularity threshold is selected as a candidate viewpoint, thus obtaining a set of candidate viewpoints corresponding to the slice.

8. A viewing angle prediction device, characterized in that, include: The current prediction viewpoint determination module is used to determine the current prediction viewpoint corresponding to the current slice to be downloaded in the panoramic video based on the head movement trajectory of the user watching the panoramic video. The target candidate view set acquisition module is used to acquire the target candidate view set corresponding to the current slice to be downloaded if the current head turning action in the head movement trajectory is detected as an invalid action. The viewpoint correction processing module is used to determine the viewpoint difference between each target candidate viewpoint in the target candidate viewpoint set and the current predicted viewpoint, and to determine the target candidate viewpoint with the smallest viewpoint difference as the target predicted viewpoint corresponding to the current slice to be downloaded.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the view prediction method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the view prediction method as described in any one of claims 1-7.

Images