Star flash-Polar code enhanced AR / VR panoramic video stream transmission method

Abstract

The invention discloses a star flash-Polar code enhanced AR / VR panoramic video stream transmission method, and relates to the technical field of video stream transmission. According to the star flash-Polar code enhanced AR / VR panoramic video stream transmission method, panoramic video data are acquired, a gaze thermodynamic diagram is analyzed, an image is divided into a plurality of blocks, a spatial gaze priority index of each block is extracted, a differentiated Polar coding rate is allocated based on the indexes, and a star flash protocol frequency band resource state is combined, so that an AR / VR panoramic video stream is transmitted. According to the method, the dynamic division and key block identification of the panoramic image are realized by constructing the gaze thermodynamic diagram and extracting the spatial gaze priority index, and the definition and detail retention of the gaze area image are enhanced in combination with differential Polar coding rate allocation, so that the multi-link efficient transmission is realized. And the immersion and interaction experience in an AR / VR scene are improved, and the method is suitable for high-immersion panoramic application.

Description

Technical Field

[0001] The present invention relates to the technical field of video stream transmission, and in particular to a method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar codes. Background Art

[0002] Amid the rapid development of AR (augmented reality) and VR (virtual reality) technologies, panoramic video, as a core medium for immersive experiences, has been widely adopted in gaming, distance learning, medical simulation, and virtual tourism. Panoramic videos typically feature ultra-high resolution, a wide field of view, and a high frame rate. Their transmission demands extremely high bandwidth, latency, and stability. In user interaction scenarios, such as rapid head movement, intra-frame response consistency and reliability of key viewports are particularly critical. Traditional wireless transmission technologies (such as Wi-Fi and Bluetooth) often face bandwidth bottlenecks and severe interference in highly concurrent and complex link environments, making them unable to meet the low-latency, high-quality transmission requirements of panoramic videos. The SparkLink protocol, an emerging short-range, high-speed wireless communication protocol, offers multi-link parallelism, high-frequency bandwidth utilization, and low latency, providing new technical support for AR / VR panoramic video transmission. Furthermore, Polar coding, a forward error correction scheme operating at near-capacity limits, can improve transmission robustness while maintaining coding efficiency.

[0003] Limitations of existing technologies include at least the following: They fail to manage and prioritize regional content within video images based on user perspective behavior. In actual use, user gaze behavior in AR / VR devices is highly concentrated and dynamically changing. The user's gaze area at any given time often only occupies a very small portion of the panoramic image. However, traditional encoding and transmission mechanisms typically treat the entire frame as a unified entity, ignoring the significant differences in perceived value and reliability requirements between gazed and non-gazed areas. This approach can easily result in network transmission resources being evenly distributed across areas of low perceived value, making it difficult for high-value gaze areas to obtain adequate coding protection and transmission guarantees, leading to distortion or frame loss of critical content. Furthermore, in complex scenarios with limited frequency band resources and sudden link interference, the overall transmission system lacks a flexible adjustment mechanism, making it difficult to dynamically allocate resources based on user behavior. This ultimately leads to insufficient user experience stability and a significant decrease in user perceived quality, limiting the coding efficiency and transmission strategy adaptability of AR / VR panoramic video systems in highly immersive interactive scenarios. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a StarFlash-Polar code enhanced AR / VR panoramic video stream transmission method, which solves the problem that the existing technology does not perform block recognition and differentiated encoding transmission of panoramic video image content according to user gaze behavior, resulting in easy distortion of key viewing areas, low transmission efficiency, and unstable user perceived quality.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A Starflash-Polar code-enhanced AR / VR panoramic video stream transmission method includes the following steps: obtaining AR / VR panoramic video data to be transmitted, including several frames of panoramic video images; performing a view area heat analysis on the AR / VR panoramic video data to obtain a gaze heat map for each panoramic video frame; dividing each panoramic video frame into several image blocks based on the gaze heat map, and analyzing the spatial gaze priority index of each image block in each panoramic video frame; allocating a Polar coding bit rate to each image block in each panoramic video frame based on the spatial gaze priority index; constructing a matrix mapping model between image blocks and frequency band resources based on the multi-link frequency band resource status of the Starflash protocol, and after completing the Polar coding, performing multi-link transmission of each image block on the corresponding frequency band.

[0007] Furthermore, the specific steps for obtaining the gaze heat map of each frame of panoramic video image are as follows: obtaining the user head motion data and user eye tracking data corresponding to each frame of panoramic video image, the user head motion data including the user's head rotation angle and head offset, and the user eye tracking data including the user's gaze point coordinates; based on the head rotation angle, constructing the user's rotational viewing angle projection area in each frame of panoramic video image, and adjusting the area position based on the head offset to obtain the user's gaze area in each frame of panoramic video image; based on the user's gaze point coordinates, performing coordinate mapping processing on the user's gaze area in each frame of panoramic video image to obtain the user's gaze point panoramic coordinates in each frame of panoramic video image; and constructing the gaze heat map of each frame of panoramic video image with the panoramic coordinates of the gaze point as the heat center.

[0008] Furthermore, the panoramic video image includes pixel values ​​and pixel coordinates of a plurality of pixel points, the gaze area includes a plurality of gaze pixel points, and the gaze heat map includes heat values ​​of a plurality of gaze pixel points.

[0009] Furthermore, the specific steps of constructing the gaze heat map of each frame of panoramic video image with the panoramic coordinates of the gaze point as the heat center are as follows: based on the pixel values ​​of several gaze pixels in the user's gaze area in each frame of panoramic video image, analyze the pixel mean and texture complexity of the user's gaze area in each frame of panoramic video image; based on the pixel mean and texture complexity, analyze the gaze heat diffusion range value of the user's gaze area in each frame of panoramic video image; analyze the Euclidean distance values ​​of several gaze pixels in the user's gaze area in each frame of panoramic video image and the heat center respectively, and combine with the gaze heat diffusion range value to analyze the heat value of each gaze pixel in the user's gaze area in each frame of panoramic video image.

[0010] Furthermore, based on the gaze heat map, the specific steps of dividing each frame of panoramic video image into several image blocks are as follows: dividing each frame of panoramic video image into several initial image blocks based on a preset size; for each frame of panoramic video image, reading the heat values ​​of all pixels in each initial image block respectively, and analyzing the average heat value; marking the initial image blocks with an average heat value higher than a preset heat threshold as high-heat initial image blocks; performing a re-division process on the high-heat initial image blocks to obtain several high-heat image blocks; marking the initial image blocks with an average heat value lower than or equal to the preset heat threshold as normal-heat initial image blocks, and performing a merging operation to obtain several normal-heat image blocks.

[0011] Furthermore, the specific steps for analyzing the spatial gaze priority index of each image block of each frame of panoramic video image are as follows: feature analysis is performed on several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image, and a gaze feature set of several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image is obtained, including heat mean, center deviation distance, and maximum heat gradient; based on the gaze feature set, the spatial gaze priority index of several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image is analyzed respectively.

[0012] Furthermore, the specific formula for calculating the spatial gaze priority index of a high-heat image block in a frame of panoramic video image is as follows:

[0013]

[0014] Among them, KjZ is the spatial gaze priority index of a high-heat image block of a certain frame of panoramic video image, RdJ is the heat mean of a high-heat image block of a certain frame of panoramic video image, μ1 is the heat response coefficient stored in the database, ZpJ is the center deviation distance of a high-heat image block of a certain frame of panoramic video image, μ2 is the deviation response coefficient stored in the database, RtD is the maximum heat gradient of a high-heat image block of a certain frame of panoramic video image, μ3 is the gradient response coefficient stored in the database, and η is the edge suppression coefficient stored in the database.

[0015] Furthermore, the specific steps of allocating the Polar coding bitrate to each image block of each frame of panoramic video image based on the spatial gaze priority index are as follows: the spatial gaze priority indexes of several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image are respectively compared with several preset spatial gaze priority intervals, and each spatial gaze priority interval corresponds to a Polar coding bitrate; the Polar coding bitrate corresponding to the spatial gaze priority index in the preset spatial gaze priority interval is used as the Polar coding bitrate of the corresponding heat image block.

[0016] Furthermore, based on the multi-link frequency band resource status of the Star Flash protocol, a matrix mapping model of image blocks and frequency band resources is constructed, and after completing Polar encoding, the specific steps of executing multi-link transmission of each image block on the corresponding frequency band are as follows: obtaining the real-time frequency band status data of each frequency band in the current Star Flash protocol communication system, and combining the spatial gaze priority indicators and encoding bit rate fields of several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image to construct a mapping relationship matrix between image blocks and frequency band resources; after completing Polar encoding of all image blocks of each frame of panoramic video image, based on the mapping relationship matrix between image blocks and frequency band resources, each block of each frame of panoramic video image is sent to the corresponding frequency band link.

[0017] Furthermore, by combining user head motion prediction data with image decoding feedback data, the coding resource allocation and frequency band mapping strategy for each image block are dynamically optimized. The specific steps are as follows: the user head motion prediction data corresponding to each frame of panoramic video image is obtained, and the image blocks that may become the location of the gaze point in the next frame of the image are identified and marked as predicted gaze blocks. The redundancy injection ratio of such image blocks is increased during Polar encoding; based on the decoding feedback information of the image return end, the transmission delay value and frame loss rate of each image block are obtained, and the Polar coding resource configuration and frequency band mapping priority are adjusted.

[0018] The present invention has the following beneficial effects:

[0019] 1) This method constructs an accurate gaze heatmap by acquiring the user's head rotation angle, offset, and eye gaze point coordinates for each frame of panoramic video. Based on this heatmap, the panoramic image is dynamically divided into high-heat blocks and normal-heat blocks. In the subsequent encoding stage, gaze features such as the heat mean, gaze center deviation distance, and heat gradient of each image block are further analyzed, and a spatial gaze priority index is constructed to accurately distinguish between key viewing areas and ordinary areas. Based on this index, differentiated Polar coding bitrate allocation is performed, resulting in a higher redundancy ratio and image quality assurance for the gaze area. This processing method can significantly improve image clarity and detail retention in the user's gaze area when the user actually views AR / VR content, avoid the key image blur and distortion caused by traditional average bitrate compression, and enhance user immersion and interactive experience. It is particularly suitable for highly immersive scenarios such as virtual exhibitions, distance education, and panoramic interactive video applications.

[0020] 2) The solution of the present invention constructs a matrix mapping model between image blocks and frequency band resources, combined with the multi-link communication capabilities supported by the Star Flash protocol, to achieve intelligent scheduling of image blocks between frequency band links. By acquiring parameters such as the remaining bandwidth value, link interference strength, idle ratio, and priority level of each frequency band in real time, and combining the spatial gaze priority index and encoding bitrate field of each image block, a dynamic mapping relationship is established to ensure that high-attention-priority image blocks are preferentially allocated to high-quality links. At the same time, after Polar encoding is completed, each image block is sent to the corresponding link according to the mapping result, ensuring the stability and low latency of important content transmission. In addition, the system also combines image return decoding feedback data to obtain block-level transmission delay and frame loss rate information, and adjusts the frequency band mapping priority strategy in real time, thereby achieving feedback-driven transmission path optimization. This effectively solves the problems of screen freeze and key frame delay in traditional panoramic video in high-load wireless environments, and improves overall transmission efficiency and reliability.

[0021] 3) The present invention introduces user head motion prediction data and decoding feedback information from the image return end to construct a resource allocation optimization mechanism based on the linkage between user interaction behavior and system feedback. In the pre-encoding stage, the system identifies potential next-frame gaze blocks based on the user's head posture prediction results and marks them as predicted gaze blocks, significantly improving the Polar redundancy injection ratio of such blocks and preventing image loading blur caused by sudden changes in perspective. In the post-encoding stage, the system uses the delay value and frame loss rate data from the return decoding end to fine-tune the current block-level Polar code rate and frequency band allocation strategy, thereby constructing a dynamic closed-loop optimization system. This mechanism effectively avoids the resource waste and response delay problems existing in traditional static coding strategies, ensures that coding resources are always focused on the user's attention area, and flexibly adapts and adjusts according to the actual transmission situation, reflecting the high adaptability, high intelligence and scenario adaptability of the present invention in transmission resource allocation strategies.

[0022] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the StarFlash-Polar code enhanced AR / VR panoramic video stream transmission method of the present invention.

[0024] Figure 2 This is a flowchart of the specific steps of dividing each frame of panoramic video image into several image blocks in the Star Flash-Polar code enhanced AR / VR panoramic video stream transmission method of the present invention.

[0025] Figure 3 This is a flowchart of the specific steps for analyzing the spatial gaze priority index of each image block in each frame of panoramic video image in the Star Flash-Polar code enhanced AR / VR panoramic video stream transmission method of the present invention. DETAILED DESCRIPTION

[0026] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0027] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0028] The term "comprising" herein indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. The terms "comprising", "including", "having" and their variations all mean "including but not limited to", excluding that which is specifically emphasized in other ways. Hereinafter, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating one or more of the features. In the description of the embodiments of the present application, unless otherwise stated, the meaning of "plurality" is two or more.

[0029] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0030] See also Figure 1 , an embodiment of the present invention provides a Starflash-Polar code enhanced AR / VR panoramic video stream transmission method, comprising the following steps: obtaining AR / VR panoramic video data to be transmitted, including several frames of panoramic video images; performing a view area heat analysis on the AR / VR panoramic video data to obtain a gaze heat map of each frame of the panoramic video image; dividing each frame of the panoramic video image into several image blocks based on the gaze heat map, and analyzing a spatial gaze priority index of each image block of each frame of the panoramic video image; allocating a Polar coding bit rate to each image block of each frame of the panoramic video image based on the spatial gaze priority index; constructing a matrix mapping model between image blocks and frequency band resources based on the multi-link frequency band resource status of the Starflash protocol, and performing multi-link transmission of each image block on the corresponding frequency band after completing the Polar coding.

[0031] Specifically, the specific steps of obtaining the gaze heat map of each frame of the panoramic video image are as follows: obtaining the user's head motion data and user eye tracking data corresponding to each frame of the panoramic video image, the user's head motion data including the user's head rotation angle and head offset, and the user's eye tracking data including the user's gaze point coordinates;

[0032] Based on the head rotation angle, the user's rotational viewing angle projection area in each frame of the panoramic video image is constructed, and the area position is adjusted based on the head offset to obtain the user's gaze area in each frame of the panoramic video image, which is specifically:

[0033] First, read the user's head rotation angle corresponding to each frame of panoramic video image, including pitch angle (indicating the angle of the user's head rotating up and down), yaw angle (indicating the angle of the user's head rotating left and right), and roll angle (indicating the angle of the user's head tilt). The data of these angles comes from the head-mounted device, indicating the rotation of the user's head in three-dimensional space, and calculate the user's perspective projection area. The perspective projection area refers to the image range actually seen by the user after rotating the head. In the panoramic image, the user's perspective projection area is determined by the rotation angle and the original field of view of the image. Taking the yaw angle as an example, when the user's head yaw angle changes, the perspective will shift horizontally, and the gaze area in the image will shift to the left or right. Similarly, The change in pitch angle will change the up and down position of the field of view, and the roll angle will change the tilt of the image. After determining the rotated viewing area, the spatial offset of the user's head will also affect the position of the actual gaze area. The spatial offset refers to the change in the position of the user's head (for example, moving left or right, up or down). By obtaining the lateral offset (i.e., the change in the horizontal position of the head) and the longitudinal offset (i.e., the change in the vertical position of the head) of the user's head, the rotated viewing area is translated and adjusted to keep it consistent with the actual head position of the user. Finally, the user's gaze area in each frame of the panoramic video image is obtained. This area represents the part of the image that the user actually gazes at after the head is rotated and offset.

[0034] Based on the user's gaze point coordinates, coordinate mapping processing is performed on the user's gaze area in each frame of the panoramic video image to obtain the panoramic coordinates of the user's gaze point in each frame of the panoramic video image. Specifically, the user's eye tracking data is read, including the gaze point coordinates, that is, the image position where the user's eyes are looking. The gaze point coordinates are two-dimensional and represent the specific position of the image where the user is looking at at the current moment. This position is obtained by the eye tracking device and is relative to the user's field of view coordinate system. Next, the gaze point coordinates are mapped to the rotated perspective projection area. Since the rotation of the user's head changes the gaze area of ​​the image, the gaze point coordinates in the eye tracking data need to be converted from the user's field of view coordinate system to the rotated perspective coordinate system. Then, coordinate mapping is performed to map the relative position of the gaze point in the rotated perspective projection area back to the global coordinate system of the panoramic image. This process involves converting the gaze point coordinates according to the current viewing angle range and image size ratio, thereby determining the accurate position of the gaze point in the panoramic image. Finally, the panoramic coordinates of the user's gaze point in each frame of the panoramic video image are obtained, which represent the specific position of the user's gaze in the current image.

[0035] Taking the panoramic coordinates of the gaze point as the heat center, a gaze heat map of each frame of panoramic video image is constructed.

[0036] In this implementation, by combining the user's head rotation angle and spatial offset, the rotational perspective projection area is accurately constructed, and the gaze point coordinates in the eye tracking data are introduced to achieve accurate mapping of the gaze area and the gaze point in the panoramic image, thereby generating a gaze heat map that truly reflects the user's gaze behavior. This heat map can not only dynamically reflect the user's gaze focus at different time points, but also provide detailed spatial positioning information, providing a high-precision basis for subsequent image coding resource allocation and transmission priority determination, significantly improving the coding efficiency and transmission guarantee capabilities of key views, and effectively supporting the optimization of user perception experience in high-bandwidth AR / VR application scenarios.

[0037] Specifically, the panoramic video image includes pixel values ​​and pixel coordinates of a plurality of pixel points, the gaze area includes a plurality of gaze pixel points, and the gaze heat map includes heat values ​​of a plurality of gaze pixel points.

[0038] The specific steps for constructing the gaze heat map of each frame of panoramic video image with the panoramic coordinates of the gaze point as the heat center are as follows:

[0039] Based on the pixel values ​​of several fixation pixels in the user's fixation area in each frame of panoramic video, the pixel mean and texture complexity of the user's fixation area in each frame of panoramic video are analyzed. Specifically, the fixation area determined by the user's eye tracking data is extracted from each frame of panoramic video. The fixation area is a two-dimensional rectangular area with a side length of 80×80 pixels. The center of the area is the panoramic coordinates of the user's fixation point in the current frame of the image. The grayscale values ​​of all pixels in the fixation area are read in the grayscale value range [0, 255]. The grayscale values ​​of all pixels in the fixation area are accumulated and divided by the total number of pixels (6400) to obtain the pixel brightness mean of the fixation area. Subsequently, a Sobel edge detection operation is performed on the fixation area. The gradient change value in the horizontal and vertical directions is calculated for each pixel. The number of pixels with a gradient value greater than a set threshold (e.g., 20) is counted to obtain the number of texture edges in the fixation area. This number is divided by 6400 to obtain the texture complexity of the area, expressed as a decimal value between 0 and 1. Higher texture complexity indicates richer details.

[0040] Based on the pixel mean and texture complexity, the gaze heat diffusion range value of the user's gaze area in each frame of the panoramic video image is analyzed, which is specifically:

[0041] According to the calculated pixel brightness mean and texture complexity value, a linear mapping method is used to construct the gaze heat diffusion range value.

[0042] Specifically, if the mean pixel brightness is greater than 200, it means that the brightness of the user's gaze area is high, and the value is mapped to the diffusion influence coefficient α1=0.8; if the brightness value is less than 100, it is mapped to α1=0.5, and other values ​​are obtained by proportional linear interpolation. If the texture complexity value is greater than 0.5, it means that the structure of the user's gaze area is complex, and the value is mapped to the diffusion influence coefficient α2=0.9; if the texture complexity is less than 0.2, it is mapped to α2=0.4. Then, by combining the above two influence coefficients, the gaze heat diffusion range value σ is obtained. The calculation example formula is σ=10×(0.6×α1+0.4×α2). The final σ value represents the diffusion radius of the Gaussian heat function. The larger the value, the wider the heat influence range. The σ value will be used to assign heat weights to the surrounding pixels with the gaze point as the center.

[0043] Analyze the Euclidean distance values ​​of several gaze pixels in the user's gaze area in each frame of panoramic video image and the heat center, and combine them with the gaze heat diffusion range value to analyze the heat value of each gaze pixel in the user's gaze area in each frame of panoramic video image.

[0044] The specific formula for calculating the heat value of a certain gaze pixel point in the user's gaze area in a certain frame of panoramic video image is as follows:

[0045] Among them, H(x, y) is the heat value of the gaze pixel point located at (x, y) in the user's gaze area in a certain frame of panoramic video image, (x0, y0) is the panoramic coordinates of the user's gaze point in a certain frame of panoramic video image, that is, the heat center, and σ is the gaze heat diffusion range value of the user's gaze area in a certain frame of panoramic video image.

[0046] In this implementation, the two dimensions of pixel brightness mean and texture complexity are introduced to quantitatively analyze the features of the gaze area. Combined with the Gaussian diffusion mechanism, the heat diffusion range centered on the gaze point and the heat value of each pixel are accurately calculated to construct a gaze heat map with physiological significance and visual perception correspondence. This heat map can not only reflect the true distribution of the user's attention to the image area, but also adaptively adjust the heat diffusion weight according to the image details and brightness differences, significantly improving the accuracy of gaze modeling and adaptability to individual differences, and providing a more differentiated spatial resource reference for subsequent coding optimization and frequency band scheduling.

[0047] Specifically, if Figure 2 As shown in FIG, the specific steps of dividing each frame of panoramic video image into several image blocks based on the gaze heat map are as follows:

[0048] Dividing each frame of panoramic video image into a number of initial image blocks based on a preset size;

[0049] For each frame of panoramic video, the heat values ​​of all pixels in each initial image block are read and the average heat value is analyzed. The initial image blocks with an average heat value higher than a preset heat threshold are marked as high-heat initial image blocks.

[0050] A redivision process is performed on the high-heat initial image blocks to obtain several high-heat image blocks (for example, each high-heat initial image block is redivided into 3×3 sub-blocks, and the size of each heat image block is 20×20 pixels); the initial image blocks whose average heat value is lower than or equal to the preset heat threshold are marked as normal-heat initial image blocks, and a merging operation is performed (this type of normal-heat block and its surrounding adjacent blocks are merged into one area unit) to obtain several normal-heat image blocks.

[0051] In this embodiment, by implementing a hierarchical image block division strategy on the gaze heat map, refined modeling of the user's attention area and non-attention area is achieved.

[0052] Specifically, a heat threshold is used to divide the initial image blocks into two categories: high heat and normal heat. The high heat blocks are then fine-grainedly subdivided and the normal heat blocks are adjacently merged. This not only improves the spatial resolution of the user's high-attention areas, but also effectively compresses the processing redundancy of the low-attention areas. This optimizes the overall coding granularity and resource scheduling structure while ensuring the accuracy of the visual core blocks. This method improves the sensitivity and efficiency of coding resources and frequency band allocation, providing a structural basis for dynamic transmission optimization.

[0053] Specifically, if Figure 3 As shown in FIG, the specific steps of analyzing the spatial gaze priority index of each image block of each frame of panoramic video image are as follows:

[0054] Feature analysis is performed on several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image, and a gaze feature set of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image is obtained, including heat mean, center deviation distance (Euclidean distance from the center point to the panoramic coordinates of the gaze point), and maximum heat gradient (the heat value of each pixel in the image block is read to construct a heat matrix; then the heat difference of each pixel in the horizontal and vertical directions is calculated to obtain the corresponding gradient vector, and the heat gradient amplitude of each pixel is calculated based on this; finally, all gradient amplitudes are traversed in the entire block, and the maximum value is extracted as the maximum heat gradient of the image block); based on the gaze feature set, the spatial gaze priority indicators of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image are analyzed respectively.

[0055] The specific formula for calculating the spatial gaze priority index of a high-heat image block in a panoramic video frame is as follows:

[0056] Among them, KjZ is the spatial gaze priority index of a high-heat image block of a certain frame of panoramic video image, RdJ is the heat mean of a high-heat image block of a certain frame of panoramic video image, μ1 is the heat response coefficient stored in the database, ZpJ is the center deviation distance of a high-heat image block of a certain frame of panoramic video image, μ2 is the deviation response coefficient stored in the database, RtD is the maximum heat gradient of a high-heat image block of a certain frame of panoramic video image, μ3 is the gradient response coefficient stored in the database, and η is the edge suppression coefficient stored in the database. For example, if the block is at the edge of the image, the value is 0.2, otherwise the value is 0.

[0057] It should be explained that when calculating the spatial gaze priority index of a high-heat image block in a frame of panoramic video image, the heat mean, center deviation distance, and maximum heat gradient need to be de-unitized.

[0058] The specific steps for obtaining the thermal response coefficient μ1, deviation response coefficient μ2, gradient response coefficient μ3, and edge suppression coefficient η stored in the database are as follows:

[0059] For the heat response coefficient: During the sample training phase, the fitting relationship between the heat mean value range corresponding to the user's gaze area in multiple panoramic video clips and the final gaze dwell time is collected. The least squares method is used for regression modeling. The optimal fitting coefficient of the heat mean term in the fitting function is extracted as μ1 and stored in the response coefficient database.

[0060] For the deviation response coefficient: the pixel distance between the center point of all video blocks in each frame and the user's gaze point is counted, and an inverse relationship curve is established between this distance and the probability of subsequent user interaction. The exponential decay factor of the distance term is extracted using an exponential fitting method and stored in the database as μ2;

[0061] For the gradient response coefficient: statistically analyze the frequency of changes between the grayscale gradient of pixels in the fixation area and the user's eye movement scanning path in multiple training images. Select a sample group with a significantly increased speed of user gaze changes in the high-gradient area, calculate the sensitive change ratio between the maximum thermal gradient and the eye movement saccade frequency, and average this ratio to obtain μ3;

[0062] For the edge suppression coefficient: when constructing the image block index structure, a pixel mask map is created for the edge area of ​​each frame image. For example, if the distance between the center point of a block and the image boundary is lower than the set threshold (such as within 10% of the boundary), the block is assigned η = 0.2, otherwise it is assigned 0.

[0063] In this implementation, by constructing a comprehensive gaze feature set (including heat mean, center deviation distance and maximum heat gradient), and combining it with a response coefficient database, the spatial gaze priority index of each image block is quantified, thereby achieving an accurate characterization of the user's visual attention level. The heat response coefficient reflects the dwell effect of gaze intensity, the center deviation distance reveals the spatial proximity of the block to the gaze point, the maximum heat gradient reflects the aggregation characteristics of the focus within the region, and the edge suppression factor effectively eliminates the misleading influence of the edge interference area. This indicator system is not only reasonably constructed and has a clear physical meaning, but also trainable and adaptable, providing an accurate basis for the differentiated allocation and frequency band mapping of subsequent coding resources, thereby improving the quality assurance capability of key blocks in panoramic video transmission.

[0064] Specifically, the steps for allocating the Polar encoding bitrate to each image block of each panoramic video frame based on the spatial gaze priority index are as follows:

[0065] The spatial gaze priority indices of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image are respectively compared with several preset spatial gaze priority intervals, and each spatial gaze priority interval corresponds to a Polar encoding bit rate; the Polar encoding bit rate corresponding to the spatial gaze priority index in the preset spatial gaze priority interval is used as the Polar encoding bit rate of the corresponding thermal image block, including but not limited to the following examples:

[0066] High-priority blocks (gaze priority index ≥ 0.85): High-heat image blocks near the user's gaze point, located in the center of the image, with high average heat and drastically changing gradients. Their Polar encoding bitrate is set to 1.5 Mbps, using a high-density coding mode, a redundancy injection ratio of 25%, and a 24-bit CRC extension segment to improve error tolerance.

[0067] Medium-high priority blocks (0.65 ≤ gaze priority index < 0.85): The heat is high but slightly away from the gaze point area, the edge detail complexity is medium, the Polar encoding bit rate is set to 1.2Mbps, the redundancy injection ratio is 15%, and the standard 16-bit CRC check segment is used.

[0068] Medium-priority blocks (0.45 ≤ gaze priority index < 0.65): These are adjacent blocks to which users may saccade, with average visual attention. The Polar encoding bitrate is set to 0.9 Mbps, with a redundancy injection ratio of 10% to maintain basic error correction capabilities.

[0069] Low-priority blocks (gaze priority index <0.45): These are edge background blocks far from the gaze area, with minimal changes in image heat and detail. Their Polar encoding bitrate is reduced to 0.6 Mbps, with a redundancy injection ratio of only 5%. Minimum encoding configuration is used to conserve transmission resources.

[0070] In this implementation, differentiated encoding bit rates and redundancy injection ratios are set for image blocks with different gaze levels, so that the key image blocks corresponding to the user's gaze area can obtain higher bit rates and stronger error correction capabilities during the encoding process, so that when there is jitter or bandwidth limitation in the transmission link, the transmission quality and stability of the core visual content can still be prioritized; while low-bit-rate compression is used for non-gaze areas to save resources and reduce redundant overhead. This mechanism improves the efficiency of encoding resource utilization, enhances the anti-interference ability and interactive response performance of panoramic videos in a star-flash multi-link environment, and is suitable for AR / VR application scenarios with extremely high requirements for real-time and subjective perception quality.

[0071] Specifically, based on the multi-link frequency band resource status of the StarFlash protocol, a matrix mapping model between image blocks and frequency band resources is constructed. After completing Polar encoding, the specific steps for executing multi-link transmission of each image block on the corresponding frequency band are as follows:

[0072] The real-time frequency band status data of each frequency band in the current Star Flash protocol communication system is obtained. The spatial gaze priority index and encoding bit rate field of several high-heat image blocks and normal-heat image blocks of each frame of panoramic video image are combined to construct a mapping relationship matrix between image blocks and frequency band resources. Specifically, the matrix is ​​as follows:

[0073] First, obtain the real-time band status data of all available frequency bands in the current Star Flash communication system. The band status data includes the band number, the current remaining bandwidth value (in Mbps), the link interference strength value (in dB), the band idle ratio (a decimal between 0 and 1), and the band priority level identifier. This data is collected by the link scheduling unit every 50ms.

[0074] Subsequently, the spatial gaze priority index and the corresponding Polar-encoded bitrate field of all image blocks in the current frame's panoramic video image are read. The image blocks are sorted from high to low based on the gaze priority value, and high-priority blocks are preferentially matched to frequency band resources with high idle ratios, high bandwidth, and low interference, while low-priority blocks are matched to the remaining resource bands. A mapping relationship matrix is ​​constructed for all image blocks and the selected frequency bands. Each row of the matrix records an image block number, spatial gaze priority, Polar bitrate field, and corresponding frequency band number for subsequent link scheduling module calls, achieving a one-to-one binding between image blocks and frequency band links.

[0075] After completing the Polar coding of all image blocks in each frame of panoramic video image, each block of each frame of panoramic video image is sent to the corresponding frequency band link based on the mapping relationship matrix between image blocks and frequency band resources. Specifically, after completing the Polar coding operation of all image blocks in each frame of image, the coded bit stream of each block is packetized into 8Kbit and packaged into a standard data frame. Each frame contains fields such as block number, frame sequence number, and target frequency band number. Then, the link scheduling module is called to write each data frame into the corresponding frequency band link in sequence according to the target frequency band number recorded in the aforementioned mapping matrix. The link scheduling module dynamically schedules the sending buffer area of ​​the channel according to the frequency band priority and the idle ratio to ensure that the data frames corresponding to the high-priority image blocks are sent first; if the frequency band link jitter or bandwidth drop exceeds the set threshold (such as 20%) during the transmission process, the system automatically activates the mapping matrix adjustment mechanism to rebind the affected image blocks to the backup frequency band link and rewrite the target frequency band number field in the data frame, ultimately ensuring that each image block of each frame can be transmitted in parallel and stably in the multi-band link supported by the Star Flash protocol, thereby improving the overall visual stability and intra-frame response consistency.

[0076] Among them, the frequency band number is used to identify each independent frequency band resource link in the Star Flash communication system. The system assigns a fixed number to each physical frequency band channel during the initialization phase. It is usually numbered according to the frequency allocation table of the built-in RF front-end of the communication chip, such as Band_1 (2.4GHz±XMHz), Band_2 (5.8GHz±XMHz), etc. The number is uniformly assigned by the resource management module in the communication protocol stack and serves as a unique identifier in subsequent frequency band scheduling and cannot be changed.

[0077] The current remaining bandwidth value indicates the size of the unoccupied transmission bandwidth of each frequency band in the current scheduling cycle. It is obtained as follows: the frequency band scheduling module reads the sum of the bandwidth allocated to each frequency band in the current cycle (in Mbps), and then deducts this value from the frequency band's maximum bandwidth threshold (such as 20Mbps) to obtain the remaining bandwidth. For example, if 12Mbps is currently allocated and the maximum is 20Mbps, the remaining bandwidth is 8Mbps. The system periodically updates this parameter every 100ms, and the data is stored in the scheduling status buffer for reading by the mapping module.

[0078] The link interference strength value indicates the strength of the interference signal received in the current frequency band, measured in decibels (dB). It is obtained by combining the RSSI (Received Signal Strength Indicator) and the background noise detector in the RF receiving module. The system first measures the background noise power at the receiving end of the current frequency band, then measures the total received power. The difference between the two is the link interference strength. The formula is: Interference Strength = Received Power (dBm) - Background Noise Power (dBm). The system performs interference assessment every 50ms.

[0079] The frequency band idle ratio indicates the proportion of time the frequency band is actually idle within a unit scheduling cycle (such as 100ms). It is obtained through the timeslot-level link detection mechanism. Each frequency band in the communication system operates according to the timeslot transmission mechanism. If a frequency band has 80 idle timeslots in a cycle and a total of 100 timeslots, the idle ratio is 0.8. The idle state is determined by detecting the channel energy threshold (for example, below -95dBm is considered idle). This value is automatically updated every cycle.

[0080] The frequency band priority identifier is pre-set by the system management policy, reflecting the available priority of the frequency band in scheduling. The value range is usually 1 to 5. The smaller the value, the higher the priority. The priority setting rules can be based on historical stability, bit error rate statistics, equipment compatibility, etc. For example, the frequency band with little interference and stable transmission is set to level 1, and the frequency band with frequent fluctuations is set to level 3 or lower. This value is stored in the frequency band configuration table in the protocol control module, and the mapping process matches resources according to this priority order.

[0081] In this implementation, by constructing a dynamic mapping relationship matrix between image blocks and frequency band resources, and combining the spatial attention priority index and coding bit rate field of each block, a high-precision, high-efficiency multi-link resource matching and parallel transmission mechanism is achieved. The system prioritizes matching high-quality frequency bands to high-attention blocks based on real-time frequency band status data (including remaining bandwidth, interference intensity, idle ratio and priority level) under the Star Flash protocol, thereby achieving highly reliable transmission of key content. At the same time, the Polar-encoded data frames are scheduled according to the frequency band number to improve link utilization and intra-frame synchronization, and the remapping guarantee mechanism is automatically triggered when the link is abnormal, significantly enhancing the stability of panoramic video in complex wireless environments and the user viewing experience.

[0082] Specifically, combining the user head motion prediction data and image decoding feedback data, the coding resource allocation and frequency band mapping strategy of each image block are dynamically optimized. The specific steps are as follows: obtain the user head motion prediction data corresponding to each frame of panoramic video image, and identify the image block that may become the location of the gaze point in the next frame of image, mark it as the predicted gaze block, and improve the redundancy injection ratio of such image blocks during Polar encoding. Specifically, after each frame of image is collected, extract the user head motion prediction data, that is, the rotation angle change of the user's head within the set time window (including pitch angle, yaw angle, roll angle) and the head The horizontal and vertical displacement values ​​are calculated and weighted per unit time to form a weighted predicted gaze trend vector. Starting from the panoramic coordinates of the current frame's gaze point, the forward landing point is searched in the image block division structure along the predicted trend direction. These points are marked as predicted gaze blocks. Enhanced Polar coding is performed on these blocks during encoding, specifically increasing their base bit rate to 1.35 times the original bit rate, increasing the redundancy injection ratio by 20%, and replacing the original 16-bit cyclic redundancy check segment with a 24-bit CRC extension segment to enhance the anti-interference decoding capability of these blocks in subsequent transmissions.

[0083] Based on decoding feedback from the image backhaul end, the transmission delay and frame loss rate (FLR) of each image block are obtained, and Polar coding resource allocation and frequency band mapping priority are adjusted. Specifically, the receiver records the arrival time and original transmission time of the data packet corresponding to each image block, and uses this to calculate the transmission delay (in milliseconds). It also counts the theoretical and actual number of frames received for that image block within a set time window to obtain the FLR. This FLR is then combined with the network link status of the current period to determine whether the block is a high-risk transmission block. If a block's delay exceeds 200ms or its FLR exceeds 0.05, Polar coding resources are reallocated to it. Its base bit rate is multiplied by 1 plus the product of the FLR and a compensation factor (default value 0.6). The block is then mapped to a preferred frequency band with an idle ratio greater than 0.6 and an interference strength less than -80dB in the current system. The corresponding frequency band mapping table entry is then updated to dynamically ensure that high-risk blocks receive optimal coding and transmission resources.

[0084] In this implementation, by introducing user head motion prediction data and image decoding feedback data, the Polar coding resource allocation and frequency band mapping strategy of image blocks are dynamically optimized, effectively improving the foresight and adaptability of panoramic video transmission, predicting the user's gaze direction for the next frame, identifying and predicting the gaze block in advance and enhancing its coding strength, which can significantly reduce the risk of frame loss of key content; at the same time, based on the transmission delay and frame loss rate feedback at the decoding end, high-risk image blocks are identified in time and resource reconfiguration and optimal frequency band remapping are performed, ensuring that the overall video stream maintains high stability and picture consistency under complex network conditions, thereby enhancing the user's immersive interactive experience.

[0085] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0086] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. Star Flash-Polar code enhanced AR / VR panoramic video stream transmission method, characterized by: The following steps are involved: Obtaining AR / VR panoramic video data to be transmitted, including several frames of panoramic video images; Performing viewpoint area heat analysis on AR / VR panoramic video data to obtain a gaze heat map for each frame of panoramic video image; Based on the gaze heat map, each panoramic video frame is divided into several image blocks, and the spatial gaze priority index of each image block of each panoramic video frame is analyzed; Assigning a Polar encoding bitrate to each image block of each panoramic video frame based on the spatial gaze priority index; Based on the multi-link frequency band resource status of the StarFlash protocol, a matrix mapping model between image blocks and frequency band resources is constructed. After completing Polar encoding, multi-link transmission of each image block is performed on the corresponding frequency band.

2. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 1, characterized in that: The specific steps to obtain the gaze heat map of each frame of panoramic video image are as follows: Obtaining user head motion data and user eye tracking data corresponding to each frame of the panoramic video image, wherein the user head motion data includes the user's head rotation angle and head offset, and the user eye tracking data includes the user's gaze point coordinates; Based on the head rotation angle, the user's rotational viewing angle projection area in each frame of the panoramic video image is constructed, and the area position is adjusted based on the head offset to obtain the user's gaze area in each frame of the panoramic video image; Based on the user's gaze point coordinates, coordinate mapping processing is performed on the user's gaze area in each frame of the panoramic video image to obtain the panoramic coordinates of the user's gaze point in each frame of the panoramic video image; Taking the panoramic coordinates of the gaze point as the heat center, a gaze heat map of each frame of panoramic video image is constructed.

3. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 2, characterized in that: The panoramic video image includes pixel values ​​and pixel coordinates of a plurality of pixel points, the gaze area includes a plurality of gaze pixel points, and the gaze heat map includes heat values ​​of a plurality of gaze pixel points.

4. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 3, characterized in that: The specific steps for constructing the gaze heat map of each frame of panoramic video image with the panoramic coordinates of the gaze point as the heat center are as follows: Based on the pixel values ​​of several gaze pixels in the user's gaze area in each frame of the panoramic video image, the pixel mean and texture complexity of the user's gaze area in each frame of the panoramic video image are analyzed; Based on pixel mean and texture complexity, the gaze heat diffusion range value of the user's gaze area in each frame of panoramic video image is analyzed; Analyze the Euclidean distance values ​​of several gaze pixels in the user's gaze area in each frame of panoramic video image and the heat center, and combine them with the gaze heat diffusion range value to analyze the heat value of each gaze pixel in the user's gaze area in each frame of panoramic video image.

5. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 4, characterized in that: The specific steps for dividing each panoramic video frame into several image blocks based on the gaze heat map are as follows: Dividing each frame of panoramic video image into a number of initial image blocks based on a preset size; For each frame of panoramic video image, read the heat values ​​of all pixels in each initial image block and analyze the average heat value; Marking the initial image blocks whose average heat value is higher than the preset heat threshold as high-heat initial image blocks; Performing a subdivision process on the high-heat initial image block to obtain a plurality of high-heat image blocks; Initial image blocks whose average heat value is lower than or equal to a preset heat threshold are marked as normal heat initial image blocks, and a merging operation is performed to obtain several normal heat image blocks.

6. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 1, characterized in that: The specific steps for analyzing the spatial gaze priority index of each image block of each frame of panoramic video image are as follows: Perform feature analysis on several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image, and obtain the gaze feature sets of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image, including heat mean, center deviation distance, and maximum heat gradient; Based on the gaze feature set, the spatial gaze priority indicators of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image are analyzed respectively.

7. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 6, characterized in that: The specific formula for calculating the spatial gaze priority index of a high-heat image block in a panoramic video frame is as follows: Among them, KjZ, RdJ, ZpJ, and RtD are the spatial gaze priority index, heat mean, center deviation distance, and maximum heat gradient of a high-heat image block of a certain frame of panoramic video image, respectively. μ1, μ2, μ3, and η are the heat response coefficient, deviation response coefficient, gradient response coefficient, and edge suppression coefficient stored in the database, respectively.

8. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 6, characterized in that: The specific steps for allocating Polar encoding bitrate to each image block of each panoramic video frame based on the spatial gaze priority index are as follows: The spatial gaze priority indicators of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video are compared with several preset spatial gaze priority intervals, and each spatial gaze priority interval corresponds to a Polar encoding bit rate; The Polar encoding bit rate corresponding to the spatial gaze priority index being in the preset spatial gaze priority interval is used as the Polar encoding bit rate of the corresponding thermal image block.

9. The method for transmitting AR / VR panoramic video streams enhanced by Starflash-Polar code according to claim 6, characterized in that: Based on the multi-link frequency band resource status of the StarFlash protocol, a matrix mapping model between image blocks and frequency band resources is constructed. After completing Polar encoding, the specific steps for executing multi-link transmission of each image block on the corresponding frequency band are as follows: Obtain the real-time frequency band status data of each frequency band in the current Star Flash protocol communication system, and combine the spatial gaze priority indicators and encoding bit rate fields of several high-heat image blocks and normal-heat image blocks in each frame of panoramic video image to construct a mapping relationship matrix between image blocks and frequency band resources; After completing the Polar encoding of all image blocks of each frame of panoramic video image, each block of each frame of panoramic video image is sent to the corresponding frequency band link based on the mapping relationship matrix between image blocks and frequency band resources.

10. The method for transmitting AR / VR panoramic video streams enhanced by Star Flash-Polar code according to claim 1, characterized in that: Combining user head motion prediction data with image decoding feedback data, we dynamically optimize the coding resource allocation and frequency band mapping strategy for each image block. The specific steps are as follows: Obtain the user's head motion prediction data for each frame of panoramic video, identify the image blocks that may be the gaze point in the next frame, mark them as predicted gaze blocks, and increase the redundancy injection ratio of such image blocks during Polar encoding; Based on the decoding feedback information from the image backhaul end, the transmission delay value and frame loss rate of each image block are obtained, and the Polar coding resource configuration and frequency band mapping priority are adjusted.

Images