A 360-degree video streaming transmission method and system based on perception sensitivity value

Abstract

The present invention discloses a 360-degree video stream transmission method and system based on perceptual sensitivity value, which relates to the field of video surveillance. The viewpoint movement trajectory estimation is obtained through the viewpoint prediction module, and the video space slicing module performs slicing aggregation based on regional growth based on PSV, and the visual field area is dynamically divided according to the user's attention level. The slicing bit rate adaptive module solves the user QoE, delay, and accuracy optimization problems under constraints such as bandwidth to ensure the transmission quality of the video in the area with high user attention; by reducing the bit rate allocation of the video in the area with high user attention, the rapid response of the video transmission is ensured. Based on the dynamic prediction of the viewpoint and the adaptive allocation of global resources, it is ensured that the 360-degree video transmission can display with high quality and quickly follow the changes in the user's viewpoint. The present invention is particularly suitable for 360-degree video transmission of edge devices under limited bandwidth in smart factories because of its high efficiency, resource saving, and support for interaction.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance, and in particular to a 360-degree video stream transmission method and system based on perception sensitivity values. Background Art

[0002] With the continuous development of industrial intelligence, virtual reality (VR) technology has received unprecedented attention. 360-degree video (or panoramic video, immersive video) has been widely used in digital twins, industrial video surveillance and other fields due to its immersive interactive experience and all-round stereoscopic presentation. 360-degree video converts spherical video into flat video through spherical mapping for subsequent content preprocessing such as spherical rotation and flat area compression. Its minimum resolution requirement is 4K (4096*2160), while the resolution of 360-degree videos recorded by commonly used head-mounted interactive devices such as Oculus Quest or HTC Vive Cosmos is 10K (10240*4320). Therefore, 360-degree video streaming requires higher bandwidth than traditional flat video streaming, and the complexity of the transmission system increases dramatically. How to design an efficient 360-degree video streaming method is of great significance to ensure industrial video surveillance.

[0003] In addition, the frequent and rapid changes in user perspectives require the transmission system to quickly respond to the 360-degree video display of the corresponding field-of-view (FoV). However, due to the limited network bandwidth, the 360-degree video of the user's corresponding perspective is delayed and the resolution of the 360-degree video is reduced, making it difficult to meet the user's needs for high-quality and low-latency response, resulting in a decrease in the quality of user experience (Quality-of-experience, QoE). At the same time, the video display on the monitoring player requires the detection of video interactive targets. Therefore, how to improve end-to-end latency and detection accuracy as the primary goal under the constraints of communication resources, latency, and detection accuracy, and to ensure user-perceived QoE as much as possible, is a key issue that needs to be solved.

[0004] After searching the existing literature, it was found that the most similar implementation scheme is the Chinese patent application number 201611174397.1:, named: A method and system for playing panoramic videos based on user perspective, and its specific approach is: according to the user's spatial perspective, the panoramic video within the perspective range is played. However, this scheme does not consider the video display outside the user's perspective, and allocates the same high resolution to each area within the spatial perspective, and does not effectively utilize the characteristics that the user's attention is different in different areas within the perspective range; the patent application number is: 202110501270.0, and its name is: A panoramic video display method and display device, and its specific approach is: for each panoramic video frame, the first viewpoint position and the second viewpoint position are determined, and the high-definition video blocks corresponding to at least one viewpoint position are obtained, and the remaining video blocks are displayed in low-definition rendering. This solution can ensure that the area within the field of view is displayed as a high-definition image when the second viewpoint rotates, but it cannot guarantee the high-definition display requirements for the area outside the second viewpoint when the user moves his head quickly; the patent application number is: 201911229335.X, and the name is: A method and terminal for panoramic video playback, and its specific method is: multi-layer sampling of the original panoramic video, video space block division of each layer of sampled video, and the level division according to the number of sampling times, and the first, second, and third field of view area partitions are divided into resolution levels. However, this method is equivalent to dividing the visible area of ​​the user browsing the current frame into two quality levels according to the first and second field of view areas, and setting a low resolution in the third field of view area outside the user's visible area, resulting in a jump from high resolution to low resolution when the user's perspective changes, affecting the user experience.

[0005] Most of the existing technologies divide the user's field of view into one or two areas, and assign the same resolution to each area, which does not effectively utilize the different characteristics of users' attention in different areas within the field of view; the existing technologies only assign different levels of resolution to different field of view areas, and do not consider that users jumping from high resolution to low resolution will greatly affect the user experience; most of the existing technologies focus on the user's perceived quality, and pay less attention to the visual task accuracy of the panoramic video stream after transmission, and cannot meet the high reliability and low latency requirements of industrial video surveillance; the existing technologies pay little attention to the impact of dynamic moving targets on user attention.

[0006] Therefore, technicians in this field are committed to developing a 360-degree video streaming transmission method and system based on perceptual sensitivity values. Through online dynamic prediction of perspective and adaptive allocation of global resources, it is ensured that 360-degree video transmission can be displayed with high quality and quickly follow changes in user perspective. Summary of the invention

[0007] In view of the above-mentioned defects of the prior art, the technical problems to be solved by the present invention are how to design an effective 360-degree video transmission method under limited bandwidth resources, while maximizing monitoring accuracy and minimizing end-to-end transmission delay of video streams, and improving user perceived QoE as much as possible; how to design a spatial slicing method for 360-degree video to meet the distribution of user attention in the field of view; how to quickly solve the bit rate allocation of the current 360-degree video spatial slicing.

[0008] To achieve the above object, the present invention provides a 360-degree video stream transmission method based on perceptual sensitivity value, comprising the following steps:

[0009] Step 1: Segment the 360-degree video provided by the video content provider in the time domain to obtain video clips of equal time length;

[0010] Step 2: Set i=1 to initialize system parameters;

[0011] Step 3: Detect the target in the first frame of the video clip;

[0012] Step 4: The perspective prediction module predicts the user's perspective at the next moment based on the historical user perspective trajectory;

[0013] Step 5: The video spatial segmentation module performs spatial segmentation on each video block according to the perception sensitivity value (PSV);

[0014] Step 6: The QoE monitoring module calculates the QoE of the user at the current moment, where the user QoE is the video quality within the user's field of view minus the video quality change;

[0015] Step 7: Calculate the end-to-end delay and monitor the task accuracy.

[0016] Step 8: The slice rate adaptation module performs adaptive code rate allocation on each spatial slice;

[0017] Step 9: The monitoring player module receives the bit rate allocation of each spatial slice calculated by the slice bit rate adaptive module, and renders and displays the 360-degree video required by the user's perspective in the monitoring player;

[0018] Step 10: Set i=i+1;

[0019] Step 11, determine whether i is the last video block; if so, end; otherwise, jump to step 3.

[0020] Furthermore, in step 3, the target detection algorithm includes SSD, YOLO, and Fast RCNN, which can detect the target on the video frame and mark the target detection frame.

[0021] Furthermore, in step 4, the perspective prediction module adopts an online algorithm based on ridge regression to predict the user perspective at the next moment using the historical perspectives of the previous T moments, where T is the subscript of the time period.

[0022] Further, the following steps are included:

[0023] Step 5.1: Perform fine-grained M*N grid-shaped spatial slicing on each video block. Each fine-grained spatial slicing obtained is called an atomic slicing, which is the smallest unit that can be independently encoded and decoded.

[0024] Step 5.2, calculate the perceptual sensitivity value σ of each atomic fragment;

[0025] Step 5.3, atomic slices are aggregated into effective slices; the aggregation method of atomic slices adopts bottom-up aggregation, taking the atomic slice where the center point of the target detection frame is located as the seed, performing regional growth, and obtaining effective slices; effective slices are connected sets composed of one or several atomic slices; effective slices are the final spatial slice results.

[0026] Furthermore, in step 5.2, the perceived sensitivity value σ=σ 1 +σ 2 +σ 3 ; where σ 1 is the intersection-over-union ratio of the atomic fragment and the target detection box, σ 2 is the intersection-over-union ratio of the atomic slice and the predicted user perspective obtained in step 4, σ 3 is the relative moving speed of the target relative to the user's viewpoint, where the movement of the target is estimated using Kalman filtering.

[0027] Furthermore, in step 6, the video quality within the user's field of view is the average bit rate of the valid slices within the user's field of view; the video quality change includes the time domain video quality change and the spatial domain video quality change, the time domain video quality change is the average bit rate change of the spatial slices of the current video block and the spatial slices of the previous moment, and the spatial domain video quality change is the discrete variance of the spatial slice bit rate set of the current video block.

[0028] Furthermore, in step 7, the playback waiting time is the time required to download the current video block plus the decoding time of the current video block, minus the cache occupancy and video block length when downloading the video block, and the monitoring task processing delay and monitoring task accuracy are obtained in real time from the monitoring player.

[0029] Furthermore, the step 8, adaptive bit rate allocation comprises the following steps:

[0030] Step 8.1: For all perceived sensitivity values ​​PSV greater than PSV minThe effective shard set V (pre-set minimum threshold of perceptual sensitivity value) is traversed to obtain all possible spatial sharding bit rate combinations in the effective shard set V;

[0031] Step 8.2: Calculate the bit rate for each spatial slice bit rate combination in step 8.1. where r i,j is the bit rate of the jth spatial slice in the i-th video block, B i is the currently available bandwidth for transmitting the i-th video block, M i is the total number of spatial slices of the i-th video block, |V| is the number of elements in the valid slice set V, and η is the bit rate difference between the bit rates corresponding to two adjacent bit rate levels;

[0032] Step 8.3: If r i,j =0, then jump to step 8.2; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, then jump to step 8.2; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , The objective function J is equal to the user QoE plus the monitoring task accuracy minus the end-to-end delay;

[0033] Step 8.4: Determine the current J * Is it greater than 0? If it is greater than 0, return J * , Otherwise, skip to step 8.5;

[0034] Step 8.5: For all valid slice sets V′ whose PSV is greater than 0, traverse to obtain all possible spatial slice rate combinations in the valid slice set V′;

[0035] Step 8.6: For each spatial slice rate combination in step 8.5, calculate

[0036] Step 8.7: If r i,j =0, then jump to step 8.6; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, then jump to step 8.6; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * ,

[0037] Step 8.8, return to J * ,

[0038] The present invention also provides a 360-degree video stream transmission system based on perception sensitivity value, including a video content provider and a client;

[0039] The video content provider can provide sources of 360-degree videos, including panoramic cameras, head-mounted virtual reality devices, and 360-degree video datasets;

[0040] The client includes a view prediction module, a video space slicing module, a slice bit rate adaptation module, a bandwidth monitoring module, a QoE monitoring module, and a monitoring player module; wherein the view prediction module predicts the user's view at the next moment based on the historical user view trajectory, the video space slicing module performs spatial slicing on each video block based on the perceived sensitivity value, the slice bit rate adaptation module performs adaptive bit rate allocation on each spatial slice, the bandwidth monitoring module is responsible for monitoring the available wireless bandwidth resources of the edge device, the QoE monitoring module calculates the user's QoE at the current moment, and the monitoring player module receives the bit rate allocation of each spatial slice calculated by the slice bit rate adaptation module, and renders and displays the 360-degree video required by the user's view in the monitoring player.

[0041] Furthermore, the perspective prediction module obtains an estimate of the viewpoint movement trajectory, and performs region growing-based slice aggregation based on PSV through the video space slice module, and dynamically divides the field of view area according to the user's attention level. The slice bitrate adaptation module solves the user QoE, latency, and accuracy optimization problems under bandwidth constraints.

[0042] In a preferred embodiment of the present invention, the purpose of the present invention is to provide a 360-degree video stream transmission method for industrial video surveillance, which can realize 360-degree video stream transmission on edge devices with limited resources.

[0043] To achieve the above object, the present invention adopts the following technical solutions:

[0044] The system as a whole consists of a video content provider and a client. The video content provider includes panoramic cameras, head-mounted virtual reality devices, 360-degree video data sets, etc., which can provide sources of 360-degree videos. The client includes a view prediction module, a video space slicing module, a slice bit rate adaptation module, a bandwidth monitoring module, a QoE monitoring module, and a monitoring player module. Among them, the view prediction module predicts the user's view at the next moment based on the historical user view trajectory, the video space slicing module performs spatial slicing on each video block according to the perception sensitivity value (PSV), the slice bit rate adaptation module performs adaptive bit rate allocation on each spatial slice, the bandwidth monitoring module is responsible for monitoring the available wireless bandwidth resources of the edge device, the QoE monitoring module calculates the user's QoE at the current moment, and the monitoring player module displays the 360-degree video on the player according to the bit rate allocation of each spatial slice calculated by the slice bit rate adaptation module.

[0045] A 360-degree video stream transmission method for industrial video surveillance includes the following steps:

[0046] Step 1: Segment the 360-degree video provided by the video content provider in the time domain to obtain video clips of equal time length.

[0047] Step 2: Set i=1 and initialize system parameters.

[0048] Step 3: Detect the target on the first frame of the video clip. The target detection algorithm includes SSD, YOLO, FastRCNN and other algorithms that can detect the target on the video frame and mark the target detection frame.

[0049] Step 4: The perspective prediction module predicts the user's perspective at the next moment based on the historical user perspective trajectory. The perspective prediction module uses an online algorithm based on ridge regression to predict the user's perspective at the next moment using the historical perspectives of the previous T moments.

[0050] Step 5: The video spatial slicing module performs spatial slicing on each video block according to the perception sensitivity value (PSV). The video spatial slicing module performs spatial slicing including the following steps:

[0051] Step 1: Each video block is segmented into a fine-grained M*N grid-like space. Each fine-grained space segment obtained at this time is called an atomic segment, which is the smallest unit that can be independently encoded and decoded.

[0052] Step 2 calculates the perceptual sensitivity value σ of each atomic slice.

[0053] The calculation method of the perceptual sensitivity value σ is as follows: σ = σ1 +σ 2 +σ 3 . Where σ 1 is the intersection-over-union ratio of the atomic fragment and the target detection box, σ 2 is the intersection-over-union ratio of the atomic fragment and the predicted user perspective obtained in the fourth step, σ 3 is the relative moving speed of the target relative to the user's viewpoint, where the movement of the target is estimated using Kalman filtering.

[0054] Step 3: Aggregate the atomic slices into valid slices. The atomic slice aggregation method adopts bottom-up aggregation, taking the atomic slice where the center point of the target detection frame is located as the seed, and performing region growing to obtain valid slices. The valid slice is a connected set composed of one or more atomic slices.

[0055] The effective sharding described in step 4 is the final spatial sharding result.

[0056] Step 6: The QoE monitoring module calculates the QoE of the user at the current moment. The user QoE is the video quality within the user's field of view minus the video quality change. The video quality within the user's field of view is the average bit rate of the valid slices within the user's field of view, and the video quality change includes the temporal video quality change and the spatial video quality change. The temporal video quality change is the average bit rate change of the spatial slices of the current video block and the spatial slices of the previous moment, and the spatial video quality change is the discrete variance of the spatial slice bit rate set of the current video block.

[0057] Step 7: Calculate the end-to-end delay and monitoring task accuracy. The end-to-end delay includes the playback waiting time and the monitoring task processing delay. The playback waiting time is the time required to download the current video block plus the decoding time of the current video block, minus the cache occupancy and video block length when downloading the video block. The monitoring task processing delay and monitoring task accuracy are obtained in real time from the monitoring player.

[0058] Step 8: The slice rate adaptive module performs adaptive code rate allocation on each spatial slice. The steps of the adaptive code rate allocation method are as follows:

[0059] Step 1 For all perceptual sensitivity values ​​PSV greater than PSV min The effective slice set V of (pre-set minimum perceptual sensitivity value threshold) is traversed to obtain all possible spatial slice code rate combinations in the effective slice set V;

[0060] Step 2: For each spatial slice rate combination in step 1, calculate where r i,j is the bit rate of the jth spatial slice in the i-th video block, B i is the currently available bandwidth for transmitting the i-th video block, Mi is the total number of spatial slices of the i-th video block, |V| is the number of elements in the valid slice set V, and η is the bit rate difference between the bit rates corresponding to two adjacent bit rate levels.

[0061] Step 3 If r i,j =0, then jump to step 2; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 2; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , The objective function J is equal to the user QoE plus the monitoring task accuracy minus the end-to-end delay;

[0062] Step 4: Determine the current J * Is it greater than 0? If it is greater than 0, return J * , Otherwise, skip to step 5;

[0063] Step 5: For all valid slice sets V′ whose PSV is greater than 0, traverse to obtain all possible spatial slice code rate combinations in the valid slice set V′;

[0064] Step 6 For each spatial slice rate combination in step 5, calculate

[0065] Step 7 If r i,j =0, then jump to step 6; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 6; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * ,

[0066] Step 8 Return to J * ,

[0067] Step 9: The monitoring player module displays the 360-degree video on the player according to the bit rate distribution of each spatial segment calculated by the segment bit rate adaptation module.

[0068] Step 10: Set i=i+1.

[0069] Step 11: Determine whether i is the last video block. If yes, then end; otherwise, jump to step 3.

[0070] Compared with the prior art, the present invention has the following obvious substantial features and significant advantages:

[0071] 1. The present invention obtains the viewpoint movement trajectory estimation through online perspective prediction, performs slice aggregation based on region growing based on PSV, dynamically divides the field of view area according to the user's attention level, avoids dividing the area containing the same semantic target into redundant small spatial slices, thereby improving the video quality smoothness and user QoE.

[0072] 2. By solving the optimization problems of user QoE, latency, and accuracy under constraints such as bandwidth, the transmission quality of videos in areas with high user attention is guaranteed; by reducing the bit rate allocation of videos in areas with low user attention, the rapid response of video transmission is guaranteed.

[0073] 3. Based on dynamic prediction of viewpoint and adaptive allocation of global resources, it ensures that 360-degree video transmission can display with high quality and quickly follow the changes of user viewpoint. This method is particularly suitable for 360-degree video transmission of edge devices under limited bandwidth in smart factories because of its high efficiency, resource saving and support for interaction.

[0074] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a structural schematic diagram of a 360-degree video stream transmission system for industrial video monitoring according to a preferred embodiment of the present invention;

[0076] Figure 2 It is a flow chart of a 360-degree video stream transmission method for industrial video monitoring according to a preferred embodiment of the present invention;

[0077] Figure 3 It is a flow chart of spatial slicing performed by a video spatial slicing module in a preferred embodiment of the present invention;

[0078] Figure 4 It is a schematic diagram of spatial slicing performed by a video spatial slicing module in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0080] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0081] The present invention provides a 360-degree video stream transmission method for industrial video monitoring, comprising the following steps:

[0082] Step 1: Segment the 360-degree video provided by the video content provider in the time domain to obtain video clips of equal time length.

[0083] Step 2: Set i=1 and initialize system parameters.

[0084] Step 3: Perform target detection on the first frame of the video clip.

[0085] Step 4: The perspective prediction module predicts the user's perspective at the next moment based on the historical user perspective trajectory.

[0086] Step 5: The video spatial segmentation module performs spatial segmentation on each video block according to the Perception Sensitivity Value (PSV).

[0087] Step 6: The QoE monitoring module calculates the user's QoE at the current moment. The user's QoE is the video quality within the user's field of view minus the video quality change.

[0088] Step 7: Calculate the end-to-end delay and monitor the task accuracy.

[0089] Step 8: The slice rate adaptation module performs adaptive bit rate allocation on each spatial slice.

[0090] Step 9: The monitoring player module receives the bit rate allocation of each spatial segment calculated by the segment bit rate adaptation module, and renders and displays the 360-degree video required by the user's perspective in the monitoring player.

[0091] Step 10: Set i=i+1.

[0092] Step 11: Determine whether i is the last video block. If yes, then end; otherwise, jump to step 3.

[0093] In the third step, the target detection algorithms include SSD, YOLO, Fast RCNN and other algorithms that can detect targets on video frames and mark target detection boxes.

[0094] In the fourth step, the viewpoint prediction module adopts an online algorithm based on ridge regression to predict the user's viewpoint at the next moment using the historical viewpoints of the previous T moments.

[0095] In the fifth step, the video spatial slicing module performs spatial slicing including the following steps:

[0096] Step 1: Each video block is segmented into a fine-grained M*N grid-like space. Each fine-grained space segment obtained at this time is called an atomic segment, which is the smallest unit that can be independently encoded and decoded.

[0097] Step 2 calculates the perceptual sensitivity value σ of each atomic slice.

[0098] Step 3: Aggregate the atomic slices into valid slices. The aggregation method of atomic slices adopts bottom-up aggregation, taking the atomic slice where the center point of the target detection frame is located as the seed, and performing region growing to obtain valid slices. A valid slice is a connected set composed of one or several atomic slices. The valid slice is the final spatial slice result.

[0099] In step 2, the calculation method of the perceptual sensitivity value σ is as follows: σ = σ 1 +σ 2 +σ 3 . Where σ 1 is the intersection-over-union ratio of the atomic fragment and the target detection box, σ 2 is the intersection-over-union ratio of the atomic fragment and the predicted user perspective obtained in the fourth step, σ 3 is the relative moving speed of the target relative to the user's viewpoint, where the movement of the target is estimated using Kalman filtering.

[0100] In the sixth step, the video quality in the user's field of view is the average bit rate of the valid slices in the user's field of view, and the video quality change includes the temporal video quality change and the spatial video quality change. The temporal video quality change is the average bit rate change of the spatial slices of the current video block and the spatial slices of the previous moment, and the spatial video quality change is the discrete variance of the spatial slice bit rate set of the current video block.

[0101] In step 7, the playback waiting time is the time required to download the current video block plus the decoding time of the current video block, minus the cache occupancy and video block duration when downloading the video block. The monitoring task processing delay and monitoring task accuracy are obtained from the monitoring player in real time.

[0102] In the eighth step, the steps of the adaptive bit rate allocation method are as follows:

[0103] Step 1 For all perceptual sensitivity values ​​PSV greater than PSV min The effective slice set V (pre-set minimum threshold of perceptual sensitivity value) is traversed to obtain all possible spatial slice rate combinations in the effective slice set V;

[0104] Step 2: Calculate the bit rate for each spatial slice bit rate combination in step 1 where r i,j is the bit rate of the jth spatial slice in the i-th video block, B iis the currently available bandwidth for transmitting the i-th video block, M i is the total number of spatial slices of the i-th video block, |V| is the number of elements in the valid slice set V, and η is the bit rate difference between the bit rates corresponding to two adjacent bit rate levels.

[0105] Step 3 If r i,j =0, then jump to step 2; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 2; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , The objective function J is equal to the user QoE plus the monitoring task accuracy minus the end-to-end delay;

[0106] Step 4: Determine the current J * Is it greater than 0? If it is greater than 0, return J * , Otherwise, skip to step 5;

[0107] Step 5: For all valid slice sets V′ whose PSV is greater than 0, traverse to obtain all possible spatial slice code rate combinations in the valid slice set V′;

[0108] Step 6 For each spatial slice rate combination in step 5, calculate

[0109] Step 7 If r i,j =0, then jump to step 6; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 6; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * ,

[0110] Step 8 Return to J * ,

[0111] The present invention provides a 360-degree video stream transmission system for industrial video monitoring, including a video content provider and a client:

[0112] Video content providers include panoramic cameras, head-mounted virtual reality devices, 360-degree video data sets, etc., which can provide sources of 360-degree videos.

[0113] The client includes a view prediction module, a video space slicing module, a slice bitrate adaptation module, a bandwidth monitoring module, a QoE monitoring module, and a monitoring player module. Among them, the view prediction module predicts the user's view at the next moment based on the historical user view trajectory, the video space slicing module performs spatial slicing on each video block based on the perceived sensitivity value, the slice bitrate adaptation module performs adaptive bitrate allocation on each spatial slice, the bandwidth monitoring module is responsible for monitoring the available wireless bandwidth resources of the edge device, the QoE monitoring module calculates the user's QoE at the current moment, and the monitoring player module receives the bitrate allocation of each spatial slice calculated by the slice bitrate adaptation module, and renders and displays the 360-degree video required by the user's view in the monitoring player.

[0114] The viewpoint prediction module obtains the estimated viewpoint movement trajectory, and the video space slicing module performs slicing aggregation based on region growth based on PSV. The visual field area is dynamically divided according to the user's attention level. The slicing bitrate adaptation module solves the user QoE, latency, and accuracy optimization problems under bandwidth constraints to ensure the transmission quality of the video in the area with high user attention. By reducing the bitrate allocation of the video in the area with high user attention, the rapid response of video transmission is guaranteed. Based on the dynamic prediction of the viewpoint and the adaptive allocation of global resources, the 360-degree video transmission can be displayed with high quality and quickly follow the changes in the user's viewpoint. This method is particularly suitable for 360-degree video transmission of edge devices under limited bandwidth in smart factories because of its high efficiency, resource saving, and support for interaction.

[0115] The present invention performs space slicing based on the perceptual sensitivity value and adaptively allocates the code rate of the space slicing.

[0116] The 360-degree video streaming transmission system for industrial video surveillance in this embodiment is as follows: Figure 1 As shown, the 360-degree video streaming method is as follows Figure 2 shown.

[0117] Step 1: Segment the 360-degree video provided by the video content provider in the time domain to obtain video clips with a time length of 1 second.

[0118] Step 2: Set i=1 and initialize system parameters.

[0119] Step 3: Detect the target on the first frame of the video clip. The target detection algorithm is the SSD algorithm.

[0120] Step 4: The perspective prediction module predicts the user's perspective at the next moment based on the historical user perspective trajectory. The perspective prediction module uses an online algorithm based on ridge regression to predict the user's perspective at the next moment using the historical perspectives of the previous 10 moments.

[0121] Step 5: The video spatial segmentation module performs spatial segmentation on each video block according to the perceptual sensitivity value. The process of the video spatial segmentation module performing spatial segmentation is as follows: Figure 3 As shown in the schematic diagram Figure 4 As shown, the following steps are included:

[0122] Step 1: Each video block is divided into fine-grained 6*12 grid-shaped spatial slices. Each fine-grained spatial slice obtained at this time is called an atomic slice, which is the smallest unit that can be independently encoded and decoded.

[0123] Step 2 calculates the perceptual sensitivity value σ of each atomic slice.

[0124] The calculation method of the perceptual sensitivity value σ is as follows: σ = σ 1 +σ 2 +σ 3 . Where σ 1 is the intersection-over-union ratio of the atomic fragment and the target detection box, σ 2 is the intersection-over-union ratio of the atomic fragment and the predicted user perspective obtained in the fourth step, σ 3 is the relative moving speed of the target relative to the user's viewpoint, where the movement of the target is estimated using Kalman filtering.

[0125] Step 3: Aggregate the atomic slices into valid slices. The atomic slice aggregation method adopts bottom-up aggregation, taking the atomic slice where the center point of the target detection frame is located as the seed, and performing region growing to obtain valid slices. The valid slice is a connected set composed of one or more atomic slices.

[0126] The effective sharding described in step 4 is the final spatial sharding result.

[0127] Step 6: The QoE monitoring module calculates the user's current QoE.

[0128] Step 7: Calculate the end-to-end delay and monitor the task accuracy.

[0129] Step 8: The slice rate adaptive module performs adaptive code rate allocation on each spatial slice. The steps of the adaptive code rate allocation method are as follows:

[0130] Step 1 For all perceptual sensitivity values ​​PSV greater than PSV min The valid shard set V is traversed to obtain all possible spatial sharding code rate combinations in the valid shard set V;

[0131] Step 2: Calculate the bit rate for each spatial slice bit rate combination in step 1 where r i,j is the bit rate of the jth spatial slice in the i-th video block, B iis the currently available bandwidth for transmitting the i-th video block, M i is the total number of spatial slices of the i-th video block, |V| is the number of elements in the valid slice set V, and η is the bit rate difference between the bit rates corresponding to two adjacent bit rate levels.

[0132] Step 3 If r i,j =0, then jump to step 2; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 2; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , The objective function J is equal to the user QoE plus the monitoring task accuracy minus the end-to-end delay;

[0133] Step 4: Determine the current J * Is it greater than 0? If it is greater than 0, return J * , Otherwise, skip to step 5;

[0134] Step 5: For all valid slice sets V′ whose PSV is greater than 0, traverse to obtain all possible spatial slice code rate combinations in the valid slice set V′;

[0135] Step 6 For each spatial slice rate combination in step 5, calculate

[0136] Step 7 If r i,j =0, then jump to step 6; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, jump to step 6; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * ,

[0137] Step 8 Return to J * ,

[0138] Step 9: The monitoring player module receives the bit rate allocation of each spatial segment calculated by the segment bit rate adaptation module, and renders and displays the 360-degree video required by the user's perspective in the monitoring player.

[0139] Step 10: Set i=i+1.

[0140] Step 11: Determine whether i is the last video block. If yes, then end; otherwise, jump to step 3.

[0141] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A 360-degree video stream transmission method based on perceptual sensitivity value, characterized in that: The following steps are involved: Step 1: Segment the 360-degree video provided by the video content provider in the time domain to obtain video clips of equal time length; Step 2: Set i=1 to initialize system parameters; Step 3: Detect the target in the first frame of the video clip; Step 4: The perspective prediction module predicts the user's perspective at the next moment based on the historical user perspective trajectory; Step 5: The video spatial segmentation module performs spatial segmentation on each video block according to the perception sensitivity value (PSV); Step 6: The QoE monitoring module calculates the QoE of the user at the current moment, where the user QoE is the video quality within the user's field of view minus the video quality change; Step 7: Calculate the end-to-end delay and monitor the task accuracy. Step 8: The slice rate adaptation module performs adaptive code rate allocation on each spatial slice; Step 9: The monitoring player module receives the bit rate allocation of each spatial slice calculated by the slice bit rate adaptive module, and renders and displays the 360-degree video required by the user's perspective in the monitoring player; Step 10: Set i=i+1; Step 11, determine whether i is the last video block; if so, end; otherwise, jump to step 3; The step 5 comprises the following steps: Step 5.1: Perform fine-grained M*N grid-shaped spatial slicing on each video block. Each fine-grained spatial slicing obtained is called an atomic slicing, which is the smallest unit that can be independently encoded and decoded. Step 5.2, calculate the perceptual sensitivity value σ of each atomic fragment; Step 5.3, the atomic slices are aggregated into effective slices; the aggregation method of the atomic slices adopts bottom-up aggregation, taking the atomic slice where the center point of the target detection frame is located as the seed, performing region growth, and obtaining effective slices; the effective slice is a connected set composed of one or several atomic slices; the effective slice is the final spatial slice result; In the step 5.2, the perceptual sensitivity value σ=σ1+σ2+σ3; wherein σ1 is the intersection-and-union ratio of the atomic slice and the target detection frame, σ2 is the intersection-and-union ratio of the atomic slice and the predicted user perspective obtained in the step 4, and σ3 is the relative moving speed of the target relative to the user's viewpoint, wherein the movement of the target is estimated by using Kalman filtering; The step 8, adaptive bit rate allocation comprises the following steps: Step 8.1: For all perceived sensitivity values ​​PSV greater than PSV min A valid shard set V with a preset minimum threshold of perceptual sensitivity value is traversed to obtain all possible spatial shard bit rate combinations in the valid shard set V; Step 8.2: Calculate the bit rate for each spatial slice bit rate combination in step 8.

1. where r i,j is the bit rate of the jth spatial slice in the i-th video block, B i is the currently available bandwidth for transmitting the i-th video block, M i is the total number of spatial slices of the i-th video block, |V| is the number of elements in the valid slice set V, and η is the bit rate difference between the bit rates corresponding to two adjacent bit rate levels; Step 8.3: If r i,j =0, then jump to step 8.2; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, then jump to step 8.2; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , The objective function J is equal to the user QoE plus the monitoring task accuracy minus the end-to-end delay; Step 8.4: Determine the current J * Is it greater than 0? If it is greater than 0, return J * , Step 8.5: For all valid slice sets V′ whose PSV is greater than 0, traverse to obtain all possible spatial slice rate combinations in the valid slice set V′; Step 8.6: For each spatial slice rate combination in step 8.5, calculate Step 8.7: If r i,j =0, then jump to step 8.6; otherwise, determine whether the current bit rate allocation meets the end-to-end delay and minimum accuracy requirements of the monitoring player. If not, then jump to step 8.6; if the end-to-end delay requirements and minimum accuracy requirements are met and J ≥ J * , then let J = J * , Step 8.8, return to J * , 2. The 360-degree video stream transmission method based on perceptual sensitivity value according to claim 1, characterized in that: In step 3, the target detection algorithm includes SSD, YOLO, and Fast RCNN, which can detect the target on the video frame and mark the target detection frame.

3. The 360-degree video stream transmission method based on perceptual sensitivity value according to claim 1, characterized in that: In step 4, the perspective prediction module adopts an online algorithm based on ridge regression to predict the user perspective at the next moment using the historical perspectives of the previous T moments, where T is the subscript of the time period.

4. The 360-degree video stream transmission method based on perceptual sensitivity value according to claim 1, characterized in that: In step 6, the video quality within the user's field of view is the average bit rate of the valid slices within the user's field of view; the video quality change includes the time domain video quality change and the spatial domain video quality change. The time domain video quality change is the average bit rate change of the spatial slices of the current video block and the spatial slices of the previous moment, and the spatial domain video quality change is the discrete variance of the set of spatial slice bit rates of the current video block.

5. The 360-degree video stream transmission method based on perceptual sensitivity value according to claim 1, characterized in that: In step 7, the playback waiting time is the time required to download the current video block plus the decoding time of the current video block, minus the cache occupancy and video block length when downloading the video block, and the monitoring task processing delay and monitoring task accuracy are obtained in real time from the monitoring player.

6. A 360-degree video streaming system based on perceptual sensitivity values ​​using the method according to any one of claims 1 to 5, characterized in that: Including video content provider and client; The video content provider can provide sources of 360-degree videos, including panoramic cameras, head-mounted virtual reality devices, and 360-degree video datasets; The client includes a view prediction module, a video space slicing module, a slice bit rate adaptation module, a bandwidth monitoring module, a QoE monitoring module, and a monitoring player module; wherein the view prediction module predicts the user's view at the next moment based on the historical user view trajectory, the video space slicing module performs spatial slicing on each video block according to the perceptual sensitivity value, the slice bit rate adaptation module performs adaptive bit rate allocation on each spatial slice, the bandwidth monitoring module is responsible for monitoring the available wireless bandwidth resources of the edge device, the QoE monitoring module calculates the QoE of the user at the current moment, and the monitoring player module receives the bit rate allocation of each spatial slice calculated by the slice bit rate adaptation module, and renders and displays the 360-degree video required by the user's view in the monitoring player.

7. The 360-degree video streaming transmission system based on perceptual sensitivity value as claimed in claim 6, characterized in that: The view prediction module obtains the viewpoint movement trajectory estimation, and performs region growing-based slice aggregation based on PSV through the video space slice module, and dynamically divides the field of view area according to the user's attention level. The slice bit rate adaptation module solves the user QoE, delay and accuracy optimization problems under bandwidth constraints.

Images