Video pre-caching method and device based on user interest points and electronic equipment
By using a video pre-caching method based on user interest points to dynamically adjust the caching priority of video blocks, the loading latency problem during FOV view switching in panoramic video caching methods is solved, achieving higher prediction accuracy and smoothness.
Patent Information
- Application Number
- CN202511450508.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-11
- Publication Date
- 2026-02-03
AI Technical Summary
Existing panoramic video caching methods have low accuracy, resulting in loading delays when switching FOV perspectives.
By determining the reference cache priority of video blocks based on video content and user's historical viewing angle data, and combining the current user's field of view and head-turning behavior prediction information, the target cache priority of video blocks is dynamically adjusted to achieve ordered caching of video blocks.
It improves the accuracy of FOV perspective prediction, reduces the loading latency of high-definition images within the FOV after the user turns their head, and enhances the smoothness of video playback and user experience.
Smart Images

Figure CN121462789A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of multimedia transmission, in particular to a video pre-caching method and device based on user interest points and an electronic device, which can be applied to an XR (Extended Reality) video watching scenario. BACKGROUND
[0002] In related technologies, a panoramic video caching method records and analyzes FOV (Field of View) request histories of all users when watching a video, predicts FOVs that the users may request at future time points through these data and FOV conversion sequences, and caches corresponding panoramic video content in advance. SUMMARY
[0003] Embodiments of the present application provide a video pre-caching method and device based on user interest points and an electronic device.
[0004] In a first aspect, embodiments of the present application provide a video pre-caching method based on user interest points, comprising: determining reference caching priorities of each video block of the video according to video content and user historical viewing angle data; adjusting the reference caching priorities of each video block according to a field of view of a current user and combining prediction information of head turning behaviors of the current user to obtain target caching priorities of each video block; caching video blocks in order according to the target caching priorities.
[0005] In a second aspect, embodiments of the present application provide a video pre-caching device based on user interest points, comprising: a determination module configured to determine reference caching priorities of each video block of the video according to video content and user historical viewing angle data; an adjustment module configured to adjust the reference caching priorities of each video block according to a field of view of a current user and combining prediction information of head turning behaviors of the current user to obtain target caching priorities of each video block; a caching module configured to cache video blocks in order according to the target caching priorities.
[0006] In a third aspect, embodiments of the present application provide an electronic device, comprising: one or more processors; The processor is configured to call instructions to cause the electronic device to perform the method of the first aspect.
[0007] In a fourth aspect, an embodiment of the present application provides a storage medium, the storage medium storing instructions, when the instructions are executed on an electronic device, causing the electronic device to perform the method of the first aspect.
[0008] In a fifth aspect, an embodiment of the present application provides a program product, including at least one of a program and instructions, when the at least one of the program and instructions is executed by an electronic device, implementing the steps of the method of the first aspect.
[0009] According to the technical solution of the present application, the area that the user is likely to watch can be effectively predicted, the content that the current user is likely to be interested in can be predicted, the direction in which the current user turns his head can be predicted in real time, and the content of the corresponding area can be cached, so that the loading time delay of the high-definition picture in the FOV after the user turns his head can be effectively reduced, thereby reducing the buffering time and improving the fluency of video playing and the user experience. Compared with the traditional pre-caching method in the related art, the present application can improve the accuracy of FOV angle prediction, and further improve the fluency of FOV angle switching.
[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS
[0011] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present application and, together with the specification, serve to explain the principles of the present application.
[0012] Figure 1 is a flowchart of a video pre-caching method based on a user interest point according to an exemplary embodiment.
[0013] Figure 2 is a flowchart of a video pre-caching method based on a user interest point according to an exemplary embodiment.
[0014] Figure 3 is an effect diagram of subject recognition according to an exemplary embodiment.
[0015] Figure 4 is an example diagram of dividing concentric ring-shaped areas according to an exemplary embodiment.
[0016] Figure 5 is an example diagram of two FOV centers having an overlapping area according to an exemplary embodiment.
[0017] Figure 6 is a flowchart of a video pre-caching method based on a user interest point according to an exemplary embodiment.
[0018] Figure 7is a flow chart of a video pre-buffering method based on a user interest point according to an exemplary embodiment.
[0019] Figure 8 is a block diagram of a video pre-buffering device based on a user interest point according to an exemplary embodiment.
[0020] Figure 9 is a block diagram of a video pre-buffering device based on a user interest point according to an exemplary embodiment.
[0021] Figure 10 is a block diagram of a device 1000 for video pre-buffering according to an exemplary embodiment. DETAILED DESCRIPTION
[0022] The embodiments of the present application are described in detail below with reference to the accompanying drawings. The same or similar components have the same reference numbers throughout the drawings and a repeated description is omitted. The embodiments described below are examples for explaining the present application and are not intended to limit the scope of the present application.
[0023] The embodiments of the present application are not exhaustive and are only schematic of some embodiments, and are not intended to limit the scope of the present application. Each step in an embodiment can be implemented as an independent embodiment without contradiction, and the steps can be combined arbitrarily, for example, a scheme after removing some steps in an embodiment can be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation in an embodiment can be combined arbitrarily. In addition, the embodiments can be combined arbitrarily, for example, some or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation of other embodiments arbitrarily.
[0024] In the embodiments of the present application, the terms and / or descriptions of the embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.
[0025] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations, and do not violate public order and good customs.
[0026] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0027] It should be noted that in the embodiments of the present application, some software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0028] In the high-definition panoramic video viewing environment, a video block transmission method based on FOV (Field of View, field of view) is often adopted. The related art panoramic video caching method can create FOV conversion according to the user's FOV interest message, including the current FOV and the FOV that may be requested at the next time point. However, the main method is to record and analyze the FOV request history of all users when watching the video, and through these data and FOV conversion sequence, predict the FOV that the user may request at the future time point, and cache the corresponding panoramic video content in advance. However, this way of caching content has low accuracy, and there is still a problem of loading delay.
[0029] Based on this, the embodiments of the present application provide a video pre-caching method and device based on user interest points and electronic equipment, which can solve the technical problem of how to improve the accuracy of FOV angle prediction to improve the fluency of FOV angle switching.
[0030] The video pre-caching method, device and electronic equipment based on user interest points of the embodiments of the present application are described below with reference to the accompanying drawings.
[0031] It should be noted that the execution subject of the video pre-caching method based on user interest points of the embodiments of the present application can be a video pre-caching device based on user interest points, which can be realized by software and / or hardware. The device can be configured in an electronic device, which can include but is not limited to a terminal, a server, etc.
[0032] Figure 1 is a flowchart of a video pre-caching method based on user interest points according to an exemplary embodiment. As Figure 1 shown, the video pre-caching method based on user interest points can include but is not limited to the following steps.
[0033] In step 101, according to the video content and the user historical viewing angle data, the reference caching priority of each video block of the video is determined.
[0034] In some embodiments, the video content can be video content of a panoramic video, for example, the video can be a high-definition panoramic video.
[0035] In some embodiments, the user historical viewing perspective data can be understood as historical viewing perspective data of the user watching the same video. Wherein, the user can include the current user and other users.
[0036] In some embodiments, the video block can be a tile block. In a possible implementation, the video can be block-encoded using FOV transmission technology, and the encoded video frame exists In an embodiment of the present application, each video block (such as a tile block) of the video can be assigned according to the video content of the video and the user historical viewing perspective data, and the reference cache priority of the video block can be determined based on the assignment of the video block.
[0037] In step 102, the reference cache priority of each video block is adjusted according to the field of view of the current user and in combination with the prediction information of the head turning behavior of the current user, and the target cache priority of each video block is obtained.
[0038] In an embodiment of the present application, the head turning behavior preference of the current user when watching the video is predicted according to the historical behavior information of the current user, and the reference cache priority of each video block is dynamically adjusted according to the head turning behavior preference of the current user, so that the target cache priority of each video block suitable for the field of view of the current user can be obtained.
[0039] For example, according to the field of view of the current user, the video blocks located in the adjacent area outside the field of view (FOV) of the current user can be set to a certain priority to consider the possible slight head turning of the current user.
[0040] In step 103, the video blocks are cached in order according to the target cache priority.
[0041] In some embodiments, the video blocks can be cached according to the size order of the target cache priority. For example, the video blocks can be cached in order from large to small according to the target cache priority.
[0042] In the above embodiments, the area that the user is likely to view can be effectively predicted, the content that the current user is likely to be interested in can be predicted, the current user's head turning direction can be predicted in real time, and the content of the corresponding area can be cached, which can effectively reduce the loading time delay of the high-definition picture in the FOV after the user turns his head, thereby reducing the buffering time and improving the smoothness of video playback and user experience. Compared with the traditional pre-caching method in the related art, the application can improve the accuracy of FOV perspective prediction, and thus improve the smoothness of FOV perspective switching.
[0043] Optionally, the reference caching priority of each video block in the video frame can be determined in combination with video content analysis and historical data analysis. In some embodiments, on the basis of any one or more of the above embodiments or combined embodiments, the video pre-caching method based on user interest points can include, but is not limited to, the following steps. Figure 2
[0044] In step 201, the video blocks of the pre-caching area are divided according to the video content, and the first caching priority of each video block is determined according to the distance of each video block from the center area of the subject in the video frame.
[0045] In some embodiments, the importance value of each video block can be set according to the distance of each video block from the center area of the subject; the importance value of each video block is normalized, and the value obtained after normalization is determined as the first caching priority of the corresponding video block. That is, the video content can be analyzed, the important areas in the video frame are identified using subject detection technology, and the pre-caching area is divided according to the importance of these areas. The video blocks in the subject area are set to have a higher caching priority, and the video blocks in the non-subject area have a lower priority.
[0046] It should be noted that the position of the subject in the panoramic video has a high caching priority, and the positions of the remaining subjects have a low caching priority. In one possible implementation, a video frame can be extracted from the panoramic video every certain period of time (e.g., 1 second), the extracted video frame can be converted into a two-dimensional planar image using ERP (Equirectangular Projection, equirectangular projection), the subject in the video frame can be extracted using image subject detection, and the position information of the subject in the planar video frame can be marked. For example, as shown in Figure 3 Each square in the figure represents a tile block, the subject in the video frame is extracted as user A using image subject detection, and the position information r of the subject in the planar video frame is marked. Optionally, since the panoramic video frame used for subject recognition is obtained after ERP projection, the image is locally stretched. In order to further improve the accuracy of subject detection, an ERP panoramic image dataset fine-tuning model can be additionally prepared to fine-tune the projected panoramic video frame, which overcomes the problem of local stretching of the image, thereby more accurately identifying and caching the video subject area.
[0047] In the embodiments of the present application, the video can be block encoded based on the FOV transmission technology. After encoding, there are MxN tile blocks in the video frame. According to the subject position information obtained by subject recognition, the importance value of the tile block within the subject range of the video frame is set, and the importance value of the tile block outside the subject range of the video frame is set to 0.
[0048] In a possible implementation, the importance value of the tile block can be set as follows: the importance value is set according to the distance of the tile block from the center area of the subject. For example, all tile blocks within the image subject area are selected, and the tile blocks within the image subject area are assigned values from the center tile block outward. These tile blocks can be divided into a subject center tile block, a subject sub-center tile block, and a subject edge tile block. The specific settings can be flexibly adjusted according to the length and width of the subject area.
[0049] For example, the width and height of the subject center tile block can be related to the width and height of all tile blocks within the subject area as follows: , According to the distribution of the subject center tile block, the tile blocks outside the circle are selected as the subject sub-center tile block, and the range of the subject sub-center tile block should not exceed the range of all tile blocks within the selected image subject area. The selection method of the subject edge tile block is the same as that of the subject sub-center tile block. The importance value can be assigned to the subject center tile block, the subject sub-center tile block, and the subject edge tile block in stages, for example, the importance value of the subject center tile block is 3 ( ), the importance value of the subject sub-center tile block is 2 ( 2), the importance value of the subject edge tile block is 1 ( ), and the importance value of other tile blocks outside the image subject area is 0 ( The importance values of these tiles were normalized to obtain the importance values of all tiles in the image frame. , , , = [1 / 2, 1 / 3, 1 / 6, 0]. For example, the value obtained after normalization can be used as the first cache priority value of this tile block.
[0050] In step 202, based on the user's historical viewing perspective data, the second cache priority of each video block is determined by analyzing its proximity to the video blocks viewed by the user.
[0051] In some embodiments, video blocks representing user viewing preferences can be obtained based on historical viewing perspective data. Using these preferred video blocks as the center, the system is divided into n concentric annular regions from the center outwards, where n is a positive integer. Video blocks within these n concentric annular regions are assigned values and normalized. The normalized values are then used as the second cache priority for the corresponding video blocks. Here, the preferred video block can be understood as a video block representing the user's preferred viewing perspective.
[0052] In embodiments of this application, the user's historical viewing perspective data may include the current user's historical viewing perspective data, and may also include the historical viewing perspective data of other users. For example, the historical viewing perspective data of all users (including the current user and other users) can be combined to calculate the second cache priority of tile blocks in a video frame by analyzing these viewing preferences.
[0053] In one possible implementation, the tile representing the user's preferred viewing angle is used as the center, dividing the area into n concentric ring regions from the center outwards. Values are assigned to the tiles within these n concentric ring regions, with all tiles at the very center of each ring being assigned a value. Assign a value of n to all tiles immediately adjacent to the next outer ring. Assigning a value of n-1 to all tiles in the outermost ring. If the value is assigned to 1, then the sequence from the inside out is [ ]=[n,n-1,...,1], where the tile blocks outside the user's preferred FOV are 0, after normalization. , ]=[ ,0].
[0054] For example, such as Figure 4The tile blocks in the four blue squares are the tile blocks in the user's preferred viewing angle. The tile blocks in the user's preferred viewing angle are divided into three concentric annular regions from the center to the outer boundary. The tile blocks in the three concentric annular regions are assigned values. All tile blocks in the innermost concentric annulus are assigned a value of 3 All tile blocks in the next outer annulus (i.e., the concentric annulus composed of green squares in FIG. 13B) are assigned a value of 2 Figure 4 All tile blocks in the outermost annulus (i.e., the concentric annulus composed of gray squares in FIG. 13B) are assigned a value of 1 All tile blocks outside the user's preferred FOV are assigned a value of 0 Figure 4 The normalized values are [1 / 2, 1 / 3, 1 / 6, 0]. Optionally, the area of each concentric annulus and the importance values of all tile blocks in the concentric annulus can be dynamically set. The normalized values are [1 / 2, 1 / 3, 1 / 6, 0]. Optionally, the area of each concentric annulus and the importance values of all tile blocks in the concentric annulus can be dynamically set.
[0055] Optionally, the value of each tile for each user's preferred viewing angle of the video block can be calculated. Then, the cumulative sum of the values assigned to each tile by historical users can be calculated and divided by the number of historical users to obtain the final importance value M[][t] of the tile block of the video frame. For example, assuming that the historical users include user A and user B, the importance value M1 of each tile block can be determined according to the tile blocks preferred by user A, the importance value M2 of each tile block can be determined according to the tile blocks preferred by user B, and the importance value M1 and the importance value M2 of each tile block can be summed and divided by the number of users to obtain the final importance value of the tile block.
[0056] It is worth noting that the value of the tile block in the center region of the FOV should be the largest, and the tile blocks in each layer of the concentric annulus (from the inner circle to the outer circle) should decrease compared to the previous layer. Otherwise, when there are multiple FOV centers, the tile blocks in the edge region will have a high cache priority (such as the overlapping region shown in FIG. 13C), which conflicts with the principle that the tile blocks in the FOV center region should be downloaded first. Figure 5 It is worth noting that the value of the tile block in the center region of the FOV should be the largest, and the tile blocks in each layer of the concentric annulus (from the inner circle to the outer circle) should decrease compared to the previous layer. Otherwise, when there are multiple FOV centers, the tile blocks in the edge region will have a high cache priority (such as the overlapping region shown in FIG. 13C), which conflicts with the principle that the tile blocks in the FOV center region should be downloaded first.
[0057] It is worth noting that the value of the tile block in the center region of the FOV should be the largest, and the tile blocks in each layer of the concentric annulus (from the inner circle to the outer circle) should decrease compared to the previous layer. Otherwise, when there are multiple FOV centers, the tile blocks in the edge region will have a high cache priority (such as the overlapping region shown in FIG. 13C), which conflicts with the principle that the tile blocks in the FOV center region should be downloaded first.
[0058] It is worth noting that the value of the tile block in the center region of the FOV should be the largest, and the tile blocks in each layer of the concentric annulus (from the inner circle to the outer circle) should decrease compared to the previous layer. Otherwise, when there are multiple FOV centers, the tile blocks in the edge region will have a high cache priority (such as the overlapping region shown in FIG. 13C), which conflicts with the principle that the tile blocks in the FOV center region should be downloaded first.
[0059] In step 203, the first cache priority and the second cache priority of each video block are fused by linear weighting to obtain a reference cache priority of each video block.
[0060] In the embodiments of the present application, the first cache priority and the second cache priority of each video block can be linearly weighted to obtain the reference cache priority of the corresponding video block. As an example, the formula of the reference cache priority of all tile blocks of a video frame is as follows:
[0061] wherein, represents the reference cache priority of each tile block in the video frame at time t, and the sequence represents the reference cache priority of each tile block, and the sequence The greater the value of the element in the sequence , the higher the priority of the corresponding tile block to be cached, and the earlier the tile block is cached. represents the first cache priority of each tile block in the video frame at time t, and the sequence represents the first cache priority of each tile block. represents the second cache priority of each tile block in the video frame at time t, and the sequence represents the second cache priority of each tile block. is a weight coefficient.
[0062] In the above embodiments, the reference cache priority of each video block is determined by linearly weighting the video content and the user historical viewing angle data, which can improve the effectiveness of the cache priority of the video block. The hierarchical importance value setting method is adopted to assign values from the main central area, the secondary central area to the edge area, and combined with the concentric ring area division, so that the cache priority is more refined and reasonable, and the accuracy of high-priority caching of key areas can be improved.
[0063] Optionally, the reference cache priority of the video block can be dynamically adjusted in combination with the field of view of the current user and the real-time prediction information of the head turning behavior of the current user. In some embodiments, on the basis of any one or more of the above embodiments or combined embodiments, as shown in Figure 6 , the video pre-caching method based on user interest points can include but is not limited to the following steps.
[0064] In step 601, the head turning behavior preference of the current user when watching the video is predicted according to the historical behavior information of the current user.
[0065] In a possible implementation, a pre-trained neural network model can be used to predict the head turning behavior preference of the current user when watching the video based on the historical behavior information of the current user. For example, the input of the neural network model can include the behavior at the previous moment (such as whether to turn the head, the turning direction, the angle of turning, and the like, and optionally, the content of the current video), and the output of the neural network model can be the head turning behavior at the next moment (such as whether to turn the head, the turning direction, the angle of turning, and the like).
[0066] In step 602, the video blocks centered on the field of view angle of the current user and the surrounding video blocks are assigned values according to the head turning behavior preference of the current user.
[0067] It is worth noting that considering that the user can slightly turn the head, priority should be given to those video blocks located in the outermost circle of the FOV of the current user. For example, the video blocks located in the outermost circle of the FOV of the current user will be assigned different values according to the relative probability of their appearance in the field of view of the current user.
[0068] For example, taking tile blocks as an example, considering that the user is more likely to turn the head to the right, the tile blocks located on the right side of the outermost circle of the FOV of the current user will be assigned a value of , the tile blocks located on the left side of the outermost circle of the FOV of the current user will be assigned a value of to reflect the second possibility; the tile blocks located on the upper side of the outermost circle of the FOV of the current user will be assigned a value of = 2; the tile blocks located on the lower side of the outermost circle of the FOV of the current user are the least likely to enter the field of view, and thus are assigned a value of = 1. And other tile blocks located in a more distant range will be assigned a value of 0, because the current user is less likely to turn the head to encounter them. In the time domain, the tile blocks in the current FOV region have an infinite priority, which is simply set to 100 here. If the user does not turn the head, the tile blocks in the current field of view need to be cached with priority at all times. After normalization, , of course the value of can be dynamically adjusted according to the historical behavior of the user. Assuming that the user has a preference for turning the head to the left when watching the video, the value of the tile blocks on the left side of the FOV of the user should be the maximum. Thus, the values of the video blocks centered on the FOV of the current user and the surrounding video blocks can be obtained.
[0069]
[0069] In step 603, the reference cache priority of each video block is adjusted according to the values of the video blocks under the field of view angle of the current user, to obtain the target cache priority of each video block.
[0070] In the embodiments of the present application, after the assignment of the video block centered on the FOV of the current user and the surrounding video blocks is obtained, the reference cache priority of each video block can be dynamically adjusted according to the assignment of each video block under the field of view angle of the current user, so that the dynamically adjusted cache strategy is adapted to the current user's perspective. For example, the reference cache priority of each video block can be adjusted by linear weighting. As an example, the adjustment of the cache priority can be represented as follows:
[0071] wherein, represents the target cache priority of each tile block in the video frame at time t, and the sequence each element in the sequence represents the target cache priority of a tile block; represents the reference cache priority of each tile block in the video frame at time t, and the sequence each element in the sequence represents the reference cache priority of a tile block; represents the assignment of each tile block in the video frame at time t under the FOV of the current user, and the sequence each element in the sequence represents the assignment of a tile block under the FOV of the current user; is a weight coefficient, .
[0072] Optionally, in some embodiments, the video blocks can be sorted in descending order according to the target cache priority, and the video blocks whose target cache priority exceeds a preset priority and whose distance from the center of the field of view angle of the current user exceeds a preset angle can be removed. For example, the target cache priority of each tile block in the video frame at time t is represented as For example, the tile blocks in the sequence can be sorted in descending order, and after sorting, they can be divided into m groups (the value of m can be set). For each group of data, if there are tile blocks whose target cache priority is very large but whose distance from the center of the FOV of the current user exceeds an angle of the field of view angle, the probability of the current user accessing these tile blocks is very small unless the current user turns his head at a large angle, so these tile blocks can be removed. In this way, the cache efficiency can be further improved and the video loading speed can be optimized.
[0073] Optionally, in some embodiments, based on the above embodiments, the present application proposes a FOV pre-cache method. For example, Figure 7As shown, when playing a video, assuming the current timestamp is t_0, the FOV perspective center (x, y, z) is determined based on the current timestamp, if there is no downloaded tile in the FOV perspective at this time, all tile blocks in the FOV are downloaded; if half of the tile blocks in the FOV perspective are cached, the remaining tile blocks are downloaded. When pre-caching, the order of tile block caching refers to the target cache priority When the tile blocks with high target cache priority are cached, the tile blocks with low target cache priority are cached. The cache threshold of the tile blocks is set , The value of the cache threshold is greater than the number of tile blocks in the FOV perspective range, The maximum value of the cache threshold is set according to the network carrying capacity, and optionally, the The value of the cache threshold is less than twice the number of tile blocks in the FOV perspective range. The cache threshold specifies that at most tile blocks with a priority of the first When the video tile block caching strategy is performed, if the time parameter is set to 2 seconds, it must be ensured that the tile blocks after 2s have been completely cached before the tile blocks before the first tile block with a timestamp. Optionally, when caching is requested, a multi-range (range request) request method can be used to request multiple segments of data from the level each time.
[0074] For example, as shown in Figure 7 When pre-caching, the order of tile block caching refers to the target cache priority , The target cache priority in the cache threshold is sorted in descending order, m tile blocks are divided into a group, the tile blocks that do not need to be downloaded are removed, and the first tile blocks after t_0 are cached. When playing the video, it is determined whether the current user triggers a head turning behavior, if the head turning behavior is not triggered, the downloaded segments in the cache are read, the current timestamp of the video is updated, and the following tile blocks are continuously cached. If it is determined that the current user triggers a head turning behavior, the current FOV perspective center of the current user is determined, and the tile blocks under the current FOV perspective are continuously downloaded and cached based on the current FOV perspective center.
[0075] In summary, the present application proposes a video block pre-caching scheme in time and space, which predicts and caches the video blocks required by the current user, reduces the loading time of the picture after the user turns his head, can effectively predict the area that the user may watch, and pre-cache the related video content, thereby reducing the buffering time and improving the smoothness of video playback and user experience.
[0076] Figure 8 is a block diagram of a user interest point based video pre-caching apparatus according to an example embodiment. As shown in Figure 8 , the user interest point based video pre-caching apparatus can include a determination module 810, an adjustment module 820 and a caching module 830.
[0077] The determination module 810 is configured to determine a reference caching priority of each video block of a video according to video content and user historical viewing perspective data.
[0078] The adjustment module 820 is configured to adjust the reference caching priority of each video block according to a current user’s field of view angle and in combination with predicted information of the current user’s head turning behavior, to obtain a target caching priority of each video block.
[0079] The caching module 830 is configured to cache the video blocks in order according to the target caching priority.
[0080] In some embodiments, the adjustment module 820 is configured to predict a head turning behavior preference of the current user when watching the video according to historical behavior information of the current user, determine a value assignment of the video block centered at the current user’s field of view angle and surrounding video blocks according to the head turning behavior preference of the current user, and adjust the reference caching priority of each video block according to the value assignment of each video block under the current user’s field of view angle, to obtain the target caching priority of each video block.
[0081] In some embodiments, the adjustment module 820 is further configured to sort the video blocks in descending order according to the target caching priority, and eliminate the video blocks whose target caching priority exceeds a preset priority and whose distance from the center of the current user’s field of view angle exceeds a preset angle.
[0082] In some embodiments, as shown in Figure 9 , based on the embodiments as shown in Figure 8 , the determination module 910 can include a first determination module 911, a second determination module 912 and a fusion module 913. The first determination module 911 is configured to divide the video blocks of the pre-caching area according to the video content, and determine a first caching priority of each video block according to a distance of each video block from a center region of a subject in the video frame. The second determination module 912 is configured to determine a second caching priority of each video block by analyzing the distance from the video block to the user’s viewing preference according to the user historical viewing perspective data. The fusion module 913 is configured to perform fusion processing on the first caching priority and the second caching priority of each video block in a linear weighting manner, to obtain the reference caching priority of each video block. In some embodiments, the first determination module 911, the second determination module 912 and the fusion module 913 can be implemented by one or more processors. Figure 9 The modules 920-930 in Figure 8 and the modules 820-830 in have the same functions and structures.
[0083] In some embodiments, the first determining module 911 is configured to: set an importance value for each video block according to the distance of the video block to the center region of the subject; and normalize the importance value of each video block, and determine the value obtained after the normalization as the first cache priority of the corresponding video block.
[0084] In some embodiments, the second determining module 912 is configured to: obtain a video block preferred by the user according to the historical viewing angle data of the user; divide the video block preferred by the user into n concentric annular regions from the inner core to the outer boundary, n being a positive integer; and assign and normalize the video blocks in the n concentric annular regions, and determine the value obtained after the normalization as the second cache priority of the corresponding video block.
[0085] As to the apparatus in the above-described embodiments, specific manners in which various modules perform operations have been described in details in embodiments of the method, and thus will not be described in details here.
[0086] Figure 10 FIG. 10 is a block diagram of an apparatus 1000 for video pre-caching according to an example embodiment. The apparatus 1000 can be, for example, an electronic device, which can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a wearable display device (such as a head-mounted device, etc.), or the like.
[0087] Referring to Figure 10 , the apparatus 1000 can include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.
[0088] The processing component 1002 generally controls the overall operations of the apparatus 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 can include one or more processors 1020 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 1002 can include one or more modules to facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 can include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.
[0089] The memory 1004 is configured to store various types of data to support operations of the device 1000. Examples of these data include instructions for any application or method operating on the device 1000, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1004 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0090] The power component 1006 provides power to the various components of the device 1000. The power component 1006 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1000.
[0091] The multimedia component 1008 includes a screen providing an output interface between the device 1000 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touch or swiping action, but also detect duration and pressure related to the touch or swiping action. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 1000 is in an operating mode, such as a photographing mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.
[0092] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC) configured to receive external audio signals when the device 1000 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1004 or transmitted via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.
[0093] The I / O interface 1012 provides an interface between the processing component 1002 and peripheral interface modules, which can be a keyboard, a click wheel, a button, and the like. These buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.
[0094] The sensor component 1014 includes one or more sensors for providing status assessments for various aspects of the device 1000. For example, the sensor component 1014 can detect an open / closed position of the device 1000, relative positioning of components of the device 1000, such as a display and keypad of the device 1000, a change in position of the device 1000 or a component of the device 1000, presence or absence of user contact with the device 1000, orientation or acceleration / deceleration of the device 1000, and temperature changes of the device 1000. The sensor component 1014 can include proximity sensor(s) configured to detect presence of nearby objects without any physical contact. The sensor component 1014 can further include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1014 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0095] The communication component 1016 is configured to facilitate wired or wireless communication between the device 1000 and another device. The device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.
[0096] In an exemplary embodiment, the device 1000 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.
[0097] In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 1004 including instructions, is also provided, which can be executed by the processor 1020 of the device 1000 to perform the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.
[0098] In an exemplary embodiment, there is also provided a program product comprising at least one of a program, instructions, which when executed by the processor 1020 of the apparatus 1000 implements the steps of the above method.
[0099] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.
[0100] It is to be understood that the application is not limited to the precise details of design or construction that have been described above and illustrated in the drawings. Various modifications and changes can be made thereunto without departing from the scope of the application. The scope of the application is indicated only by the appended claims.
Claims
1. A video pre-caching method based on user interest points, characterized in that, include: Based on the video content and the user's historical viewing perspective data, determine the reference cache priority for each video block of the video; Based on the current user's field of view and combined with the prediction information of the current user's head turning behavior, the reference cache priority of each video block is adjusted to obtain the target cache priority of each video block; Video blocks are cached sequentially according to the target cache priority.
2. The method as described in claim 1, characterized in that, The step of determining the reference cache priority for each video block based on video content and user historical viewing perspective data includes: The video content is divided into pre-cached regions, and the first cache priority of each video block is determined based on the distance between each video block and the main center region in the video frame. Based on the user's historical viewing perspective data, the second cache priority of each video block is determined by analyzing its proximity to the video blocks viewed by the user. A linear weighted method is used to fuse the first cache priority and the second cache priority of each video block to obtain a reference cache priority for each video block.
3. The method as described in claim 2, characterized in that, The step of determining the first buffer priority of each video block based on the distance between each video block and the center region of the main body in the video frame includes: An importance value is assigned to each video block based on its distance from the central region of the main body; The importance value of each video block is normalized, and the value obtained after normalization is determined as the first cache priority of the corresponding video block.
4. The method as described in claim 2, characterized in that, The step of determining the second cache priority of each video block based on the user's historical viewing perspective data by analyzing its proximity to video blocks with the user's viewing preferences includes: Based on the user's historical viewing perspective data, obtain the video blocks that the user prefers to watch; Centered on the video block of the user's viewing preference, divide it into n concentric ring regions from the innermost part to the outermost boundary, where n is a positive integer; The video blocks within the n concentric ring regions are assigned values and normalized. The values obtained after normalization are determined as the second buffer priority of the corresponding video blocks.
5. The method as described in claim 1, characterized in that, The step of adjusting the reference cache priority of each video block based on the current user's field of view and combined with the prediction information of the current user's head turning behavior to obtain the target cache priority of each video block includes: Based on the current user's historical behavior information, predict the current user's head-turning behavior preference when watching videos; Based on the current user's head-turning behavior preference, determine the values of the video block centered on the current user's field of view and the surrounding video blocks; Based on the assigned values of each video block under the current user's field of view, the reference cache priority of each video block is adjusted to obtain the target cache priority of each video block.
6. The method according to any one of claims 1-5, characterized in that, The method further includes: The video blocks are sorted in descending order according to the target cache priority, and video blocks whose target cache priority exceeds the preset priority and whose distance from the center of the current user's field of view exceeds the preset angle are removed.
7. A video pre-caching device based on user interest points, characterized in that, include: The determination module is used to determine the reference cache priority of each video block of the video based on the video content and the user's historical viewing perspective data; The adjustment module is used to adjust the reference cache priority of each video block based on the current user's field of view and combined with the prediction information of the current user's head turning behavior, so as to obtain the target cache priority of each video block. The caching module is used to cache video blocks sequentially according to the target cache priority.
8. An electronic device, characterized in that, include: One or more processors; The processor is used to invoke instructions to cause the electronic device to perform the method of any one of claims 1-6.
9. A storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1-6.
10. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the programs or instructions is executed by an electronic device, it implements the steps of the method according to any one of claims 1-6.
