Ultra-high-definition video production system and method based on 5g and vr fusion

Abstract

The application is suitable for the field of VR video technology, and provides a super high-definition video production system and method based on 5G and VR fusion, comprising the following steps: collecting VR panoramic video based on a VR camera array, and uploading the original VR panoramic video stream to cloud storage in real time through a 5G network; adopting forward error correction and adaptive code rate control in the transmission process; adopting feature point matching and an optical flow method to align multi-lens pictures, and generating seamless panoramic video; enhancing the picture quality of low-light or blurred areas, and locally performing super-resolution reconstruction on key areas; generating multiple low-resolution versions through convolution downsampling; performing immersive film review based on a VR headset, identifying quality problems of the panoramic video, and displaying visual labels in 3D space. The application constructs a cloud integrated automatic processing pipeline, and all processes from video alignment, splicing, picture quality enhancement to multi-version adaptation are completed by cloud server intelligent scheduling, and the processing is efficient.

Description

Technical Field

[0001] This invention relates to the field of VR video technology, specifically to an ultra-high-definition video production system and method based on the integration of 5G and VR. Background Technology

[0002] As media technology advances towards ultra-high definition and immersive experiences, virtual reality (VR) video has become an important form of expression in fields such as news broadcasts, sports events, and cultural performances. Traditional VR video production processes typically rely on fixed, high-speed fiber optic network environments. After capturing panoramic video using multi-lens camera arrays, the footage is physically transported to a production center, where it undergoes tedious post-production stitching and image quality optimization. This process is time-consuming and has high latency, failing to meet the demands of news events and large-scale activities requiring rapid response. Furthermore, existing review methods are still based on 2D flat screens, making it impossible to effectively assess the immersiveness and spatial coherence of 360° panoramic content.

[0003] Therefore, there is a need to provide an ultra-high-definition video production system and method based on the integration of 5G and VR, which aims to solve the above problems. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an ultra-high-definition video production system and method based on the integration of 5G and VR, so as to solve the problems existing in the above-mentioned background technology.

[0005] This invention is implemented as follows: a method for producing ultra-high-definition video based on the integration of 5G and VR, the method comprising the following steps: VR panoramic video is captured using a VR camera array and uploaded to the cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during transmission. Feature point matching and optical flow are used to align multi-lens images, automatically identify and repair stitching seams, and generate seamless panoramic videos. Image quality is enhanced in low-light or blurry areas, and local super-resolution reconstruction is performed in key areas; multiple low-resolution versions are generated through convolutional downsampling to adapt to different terminal performance. Immersive film review based on VR headset, loading virtual review space, identifying quality problems in panoramic videos, and displaying visual labels in 3D space; Using a VR controller, one can annotate films in a virtual review space and obtain interactive review reports.

[0006] As a further aspect of the present invention, the step of employing forward error correction and adaptive rate control specifically includes: The original VR panoramic video stream is cut into small data packets at predetermined time intervals, FEC encoded, the original data packets are generated, and redundancy check packets are calculated. Continuously send probe packets to the camera, calculate round-trip time, packet loss rate and jitter, and obtain the network status; Based on the network status reported by the probe, the output bitrate of the video encoder is dynamically adjusted, and the ratio of the number of original data packets to the number of redundant check packets is also dynamically adjusted.

[0007] As a further aspect of the present invention, the step of aligning multi-lens images using feature point matching and optical flow specifically includes: Based on the overlapping area images of adjacent shots, several feature points that are not affected by lighting and angle are extracted; the feature points are initially matched to find corresponding pixels in the two images, thus obtaining feature point matching pairs; Based on a deep learning optical flow algorithm, the motion vector of each pixel in the overlapping region is calculated to determine the optical flow information; The physical calibration parameters of the VR camera array are retrieved, and the image is aligned based on the physical calibration parameters, feature point matching pairs, and optical flow information.

[0008] As a further aspect of the present invention, the steps of enhancing image quality in low-light or blurred areas and performing local super-resolution reconstruction in key areas specifically include: A semantic segmentation network is used to analyze panoramic frames, identify low-light areas, blurred areas and key areas, and output a region mask for each frame to mark the areas that need to be enhanced and super-resolved. For low-light and blurred regions, a GAN-based model is used for denoising and enhancement; for critical regions, they are cropped from the mask and super-resolution is performed using a HAT model.

[0009] As a further aspect of the present invention, when automatically identifying and repairing seams, in the overlapping area, a graph cut or dynamic programming algorithm is used to determine the path with the lowest energy as the seam; multi-band fusion is performed to decompose the image into different frequency bands, wide-range feathering is performed in the low-frequency band to eliminate exposure differences, and sharp stitching is performed in the high-frequency band to preserve details.

[0010] Another objective of this invention is to provide an ultra-high-definition video production system based on the integration of 5G and VR, the system comprising: The video acquisition and transmission module is used to acquire VR panoramic video based on a VR camera array and upload the original VR panoramic video stream to cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during the transmission process. The multi-view alignment module is used to align multi-lens images using feature point matching and optical flow methods, automatically identify and repair stitching seams, and generate seamless panoramic videos. The image quality enhancement and reconstruction module is used to enhance the image quality of low-light or blurred areas and perform local super-resolution reconstruction of key areas; it generates multiple low-resolution versions through convolutional downsampling to adapt to the performance of different terminals. The immersive review module is used for immersive review based on VR headsets, loading a virtual review space, identifying quality issues in panoramic videos, and displaying visual labels in 3D space; The film review report generation module is used to annotate films in a virtual film review space using a VR controller, and generate interactive film review reports.

[0011] As a further embodiment of the present invention, the video acquisition and transmission module includes: The video slicing encoding unit is used to cut the original VR panoramic video stream into small data packets at predetermined time intervals, perform FEC encoding, generate original data packets, and calculate redundancy check packets. The network detection unit is used to continuously send detection packets to the camera, calculate round-trip time, packet loss rate and jitter, and obtain the network status. The adaptive bitrate control unit is used to dynamically adjust the output bitrate of the video encoder based on the network status feedback from the probe, and to dynamically adjust the ratio of the number of raw data packets to the number of redundant check packets.

[0012] As a further embodiment of the present invention, the multi-screen alignment module includes: The feature point matching unit is used to extract several feature points that are not affected by lighting and angle based on the overlapping area images of adjacent shots; to perform preliminary matching on the feature points, find the corresponding pixel points in the two images, and obtain feature point matching pairs; The motion vector calculation unit is used to calculate the motion vector of each pixel in the overlapping region based on the deep learning optical flow algorithm, and to determine the optical flow information. The image alignment unit is used to retrieve the physical calibration parameters of the VR camera array and perform image alignment based on the physical calibration parameters, feature point matching pairs, and optical flow information.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention ensures stable and smooth transmission of ultra-high-definition VR video streams over 5G networks through forward error correction and adaptive bitrate control strategies. It constructs a cloud-integrated automated processing pipeline, where video alignment, stitching, image enhancement, and multi-version adaptation are all intelligently scheduled by the cloud server, resulting in highly efficient processing. Through an alignment strategy combining feature point matching and optical flow, along with AI-driven image enhancement and local super-resolution technology, it not only generates seamless, high-quality panoramic videos but also specifically optimizes dark details and key area clarity, ensuring a superior viewing experience across various devices. By using VR headsets and virtual review spaces, the review process is elevated from a 2D plane to a 3D immersive environment, providing visual tags and spatial annotation functions to improve the accuracy and collaborative efficiency of review. Attached Figure Description

[0014] Figure 1 This is a flowchart of an ultra-high-definition video production method based on the integration of 5G and VR.

[0015] Figure 2 This is a flowchart of adaptive bitrate control in an ultra-high-definition video production method based on the integration of 5G and VR.

[0016] Figure 3 This is a flowchart for aligning multiple camera shots in an ultra-high-definition video production method based on the integration of 5G and VR.

[0017] Figure 4 This is a flowchart illustrating the image alignment process based on physical calibration parameters, feature point matching pairs, and optical flow information in an ultra-high-definition video production method that integrates 5G and VR.

[0018] Figure 5 This is a flowchart for image quality enhancement in an ultra-high-definition video production method based on the integration of 5G and VR.

[0019] Figure 6 This is a schematic diagram of an ultra-high-definition video production system based on the integration of 5G and VR. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0022] like Figure 1 As shown, this embodiment of the invention provides a method for producing ultra-high-definition video based on the fusion of 5G and VR, the method comprising the following steps: The S100 uses a VR camera array to capture VR panoramic video and uploads the original VR panoramic video stream to cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during transmission. The S200 uses feature point matching and optical flow to align multi-lens images, automatically identifies and repairs stitching seams, and generates seamless panoramic video. The S300 enhances image quality in low-light or blurred areas and performs local super-resolution reconstruction in key areas; it generates multiple low-resolution versions through convolutional downsampling to adapt to different terminal performance. The S400 uses a VR headset for immersive film review, loads a virtual review space, identifies quality issues in panoramic videos, and displays visual labels in a 3D space. The S500 allows users to annotate films in a virtual screening space using a VR controller, generating interactive screening reports.

[0023] In this embodiment of the invention, a 6- or 8-lens VR camera array is first used to capture VR panoramic video. Each lens supports a specific resolution, and all lenses are captured synchronously using a timecode synchronizer. A 5G CPE or module is integrated into the VR cameras, supporting SA / NSA dual-mode. This allows the original VR panoramic video stream to be uploaded to cloud storage in real time via a 5G network. It is important to note that forward error correction and adaptive bitrate control are employed during transmission to ensure stable and smooth transmission of the VR video stream over the 5G network, laying a solid foundation for remote real-time production and eliminating reliance on wired networks. After the cloud platform detects new material, it uses feature point matching and optical flow to align multiple lens images, automatically identifying and repairing seams to generate a seamless panoramic video.

[0024] Specifically, multi-band fusion is performed, decomposing the image into different frequency bands. Wide-range feathering is applied to the low-frequency band to eliminate exposure differences, while sharp stitching is performed to preserve details in the high-frequency band. The specific steps are as follows: The Laplacian pyramid fusion method is used to decompose panoramic images into a predetermined number of frequency bands to obtain low-frequency and high-frequency bands; where smooth areas of the image represent low-frequency bands and image edges and textured areas represent high-frequency bands. Wide-range feathering is applied to the decomposed low-frequency band: by smoothing the brightness gradient within the low-frequency band, the exposure differences between different lenses are eliminated, and the feathered low-frequency band is obtained. By using the low-frequency band after feathering, the exposure of the overlapping area is balanced by adjusting the feathering range and intensity, so as to eliminate the brightness difference caused by different exposure of different lenses and obtain a low-frequency band with consistent exposure. Based on the low-frequency bands with consistent exposure, the decomposed high-frequency bands are sharply stitched together to obtain the stitched high-frequency bands. Using the pixel color difference and texture feature difference in the overlapping area of ​​adjacent lenses as energy terms, a graph cut algorithm is used to optimize the stitching path of the stitched high-frequency bands: the stitching path along the lowest energy term is extended to avoid face areas and edge areas, and the texture details of the image corresponding to the stitched high-frequency bands are preserved by minimizing the gradient difference of edge pixels between adjacent lenses, thus obtaining high-frequency bands with enhanced details. A weighted average is used to fuse the low-frequency bands with consistent exposure with the high-frequency bands with enhanced details to obtain a preliminary fused panoramic image. The preliminary fused panoramic image is then subjected to overall smoothing adjustments to obtain a seamless panoramic video.

[0025] Furthermore, this invention achieves exposure balance in the low-frequency band through a frequency division processing mechanism, while employing intelligent path planning based on energy optimization in the high-frequency band, thus eliminating splicing marks while fully preserving texture details.

[0026] This invention also supports immersive review by reviewers wearing VR headsets. A virtual review space is automatically loaded into the VR headset, identifying quality issues in panoramic videos, such as still frames, black screens, and high noise. Visual labels are displayed in 3D space, hovering over the time and location of the problem in the video. Reviewers can quickly and conveniently view detailed detection data by looking at a label. Furthermore, any reviewer can use a VR controller to annotate in the 360° video space (e.g., drawing a circle to point out splicing issues) or input comments via voice (voice is transcribed into text in real-time and fixed in spatial position). All annotations and voice comments are precisely bound to the video timeline. During video playback, annotations appear at the correct time and location. After review, an interactive review report with screenshots and spatiotemporal positioning is automatically generated and sent to the production team with one click. Through VR headsets and virtual review spaces, the review process is elevated from a 2D plane to a 3D immersive environment. Reviewers can evaluate content in an immersive way, and with the help of AI-assisted visual tags and spatial annotations, they can accurately and efficiently identify and record problems, generating intuitive and interactive review reports, which greatly improves the accuracy and efficiency of review.

[0027] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of employing forward error correction and adaptive rate control specifically includes: S101, the original VR panoramic video stream is cut into small data packets at predetermined time intervals, FEC encoded, the original data packets are generated, and redundancy check packets are calculated; S102, continuously sends probe packets to the camera, calculates round-trip time, packet loss rate and jitter, and obtains network status; S103 dynamically adjusts the output bitrate of the video encoder based on the network status feedback from the probe, and dynamically adjusts the ratio of the number of original data packets to the number of redundant check packets.

[0028] In this embodiment of the invention, to ensure stable and high-quality uploading of VR video streams to the cloud in an unreliable mobile 5G network environment, the original VR panoramic video stream is first segmented into small data packets of 100ms-500ms and FEC encoded. For every K original data packets generated, M redundancy check packets are calculated, where K and M are constants. Assuming K=10 and M=3, receiving any 10 of these 13 data packets is sufficient to completely recover the original data. A network probe service is deployed in the cloud to continuously send probe packets to the camera and calculate round-trip latency, packet loss rate, and jitter to obtain the network status. Based on the network status feedback from the probe, the output bitrate of the video encoder is dynamically adjusted, and the ratio of K to M is also dynamically adjusted. For example, when the network is excellent (RTT < 50ms, packet loss rate < 0.1%): use the highest bitrate (e.g., 800Mbps) to transmit the original 8K RAW or ProRes format; when the network is good (RTT < 100ms, packet loss rate < 1%): switch to a higher compression ratio encoding (e.g., H.265 / HEVC), and reduce the bitrate to 300-500Mbps; when the network is average (packet loss rate > 2%): while reducing the bitrate, dynamically increase the FEC redundancy (e.g., increase M from 2 to 4) to use more redundant data to combat packet loss.

[0029] like Figure 3 As shown, in a preferred embodiment of the present invention, the step of aligning multi-lens images using feature point matching and optical flow specifically includes: S201, Based on the overlapping area images of adjacent shots, extract several feature points that are not affected by lighting and angle; perform preliminary matching on the feature points to find corresponding pixel points in the two images and obtain feature point matching pairs; S202, based on a deep learning optical flow algorithm, calculates the motion vector of each pixel in the overlapping region to determine the optical flow information; S203 retrieves the physical calibration parameters of the VR camera array and performs image alignment based on the physical calibration parameters, feature point matching pairs, and optical flow information.

[0030] In this embodiment of the invention, to achieve multi-camera image alignment, thousands of feature points unaffected by lighting and angle are extracted from the overlapping areas of adjacent shots using SIFT, ORB, or more advanced SuperPoint neural networks. A graph neural network-based matcher performs preliminary matching on these feature points, finding corresponding pixels in the two images to obtain feature point matching pairs. Feature point matching works well in static or slow-moving scenes, but ghosting occurs with fast-moving objects. Therefore, a deep learning optical flow algorithm is needed to calculate the motion vector of each pixel in the overlapping area to determine optical flow information, thus providing a more accurate description of pixel-level motion. Finally, the physical calibration parameters of the VR camera array are retrieved and combined with the feature point matching pairs and optical flow information to perform image alignment.

[0031] like Figure 4 As shown, in a preferred embodiment of the present invention, the step of aligning the image based on physical calibration parameters, feature point matching pairs, and optical flow information specifically includes: S2031, based on physical calibration parameters, constructs an ideal geometric model that is consistent with the camera array and eliminates lens distortion; S2032, compare all feature point matching pairs with the ideal geometric model, determine the positional differences and make adjustments, and perform static alignment; S2033, based on pixel-level motion vectors, stretches, compresses, or twists the corresponding local regions in an ideal geometric model to achieve dynamic alignment.

[0032] In this embodiment of the invention, the physical calibration parameters precisely describe the lens position and orientation, as well as the lens's inherent characteristics, such as focal length, imaging center point, and lens distortion coefficient. In virtual space, an ideal geometric model, completely identical to the real camera array and free from lens distortion, is constructed. This serves as the foundation for all subsequent alignment operations. Then, all feature point matching pairs are compared with the ideal geometric model to check the positional differences between these matching points in the actual image and the ideal model. Minor rotation and translation calculations are then performed to ensure the ideal model matches the actual shooting scene of the current frame to the greatest extent possible, achieving a rough, static alignment. The optical flow information provides the fine motion trajectory of each pixel. Through these trajectories, minor deformation adjustments such as stretching, squeezing, or twisting are made to the corresponding local areas in the ideal model, ensuring perfect alignment even for fast-moving objects at the seams. This achieves frame-by-frame, pixel-level dynamic alignment, effectively eliminating ghosting caused by object movement.

[0033] like Figure 5 As shown, in a preferred embodiment of the present invention, the steps of enhancing image quality in low-light or blurred areas and performing local super-resolution reconstruction in key areas specifically include: S301 uses a semantic segmentation network to analyze panoramic frames, identify low-light areas, blurred areas and key areas, and outputs a region mask for each frame to mark the areas that need to be enhanced and super-resolved. For S302, a GAN-based model is used for denoising and enhancement in low-light and blurred areas; for key areas, they are cropped from the mask and super-resolution is performed using a HAT model.

[0034] Specifically, for key regions, they are cropped from the mask and super-resolution is performed using the HAT model. The specific steps are as follows: A semantic segmentation network is used to process the region mask to extract the boundary coordinates and pixel range information of key regions and obtain the precise location data of the key regions; the key regions include faces, text, and the central region; Based on the precise location data of the key regions, the corresponding key region image patches that retain the original resolution and color space are cropped from the panoramic frame to obtain the initial key region image patches. Load a pre-trained HAT model to obtain a HAT model instance for super-resolution of key regions; the pre-trained HAT model is built based on a hybrid attention mechanism and can perform super-resolution reconstruction, and the parameters of the pre-trained HAT model are derived from a public dataset. The initial key region image patch is input into the HAT model instance for key region super-resolution and super-resolution processing is performed. At the same time, the texture details and edge sharpness of the image patch are enhanced through the attention mechanism to obtain high-resolution key region image patches. Local contrast adjustment and artifact suppression are performed on high-resolution key region image patches to obtain optimized super-resolution image patches; By using a full-reference image quality assessment method, the detail enhancement effects of the optimized super-resolution image patch and the initial key region image patch are compared to complete the super-resolution and obtain the super-resolution image patch.

[0035] Furthermore, by combining semantic segmentation for precise localization, HAT attention-based super-resolution reconstruction, local post-processing optimization, and full-reference quality verification, this paper establishes an efficient and reliable closed-loop processing flow, which improves image quality in key areas while ensuring detail realism and visual consistency.

[0036] Specifically, for key regions, local contrast adjustment and artifact suppression are performed on high-resolution key region image patches to obtain optimized super-resolution image patches. The specific steps are as follows: The key area is divided into local pixel blocks of a preset size, and the average brightness value and average gradient value of each local pixel block are calculated to obtain local contrast distribution data; the average gradient value is obtained by statistically analyzing the change in pixel intensity. A first threshold is determined based on the average brightness value of local pixel blocks, and a second threshold is determined based on the average gradient value of local pixel blocks. Local pixel blocks with average brightness values ​​lower than the first threshold and average gradient values ​​lower than the second threshold are classified as low-contrast regions. Gray-scale transformation is applied according to the semantic category to which the low-contrast region belongs to obtain contrast-enhanced image blocks. Specifically, the gray-scale transformation is as follows: for face regions, an S-curve is used for processing; for text regions, linear stretching is used; and for the central region, gamma correction is used. Texture features are extracted from contrast-enhanced image patches using a semantic segmentation network; artifact regions are identified by edge and texture feature matching using contrast-enhanced image patches and texture features to obtain artifact location mapping. Weak filtering and smoothing are performed on the gradient intensity of the artifact region determined by the artifact location mapping, and a transition band is constructed at the boundary between the artifact region and the normal region to eliminate local inconsistencies, so as to obtain the artifact-suppressed image patch. The artifact-suppressed image patch and the high-resolution key region image patch are used as reference image quality assessments to obtain detail preservation assessment results; based on the detail preservation assessment results, the artifact-suppressed image patch is sharpened to obtain a fine-tuned image patch; The detail enhancement effect of the fine-tuned image patch was compared with that of the high-resolution key region image patch to verify the artifact suppression results and obtain the optimized super-resolution image patch.

[0037] Furthermore, this invention establishes a differential enhancement mechanism based on semantic classification. Through quantitative evaluation, transition band smoothing, and closed-loop quality verification, it achieves a balance between detail preservation and natural transition while effectively suppressing artifacts.

[0038] Specifically, for key regions, a full-reference image quality assessment method is used to compare the detail enhancement effects of the optimized super-resolution image patch with the initial key region image patch to complete super-resolution and obtain the super-resolution image patch. The specific steps are as follows: The panoramic frame is processed by a semantic segmentation network to obtain the boundary coordinates of the key region. Using the boundary coordinates of the key region, the optimized super-resolution image patch and the initial key region image patch are spatially registered in the pixel coordinate system to ensure that the optimized super-resolution image patch and the initial key region image patch completely overlap in the pixel range of the corresponding face, text and center region, so as to obtain spatially aligned image pairs. Extract optimized super-resolution image patch texture feature vectors and initial key region image patch texture feature vectors from spatially aligned image pairs; perform point-by-point comparisons between the super-resolution image patch texture feature vectors and the initial key region image patch texture feature vectors to obtain texture matching degree parameters; Using texture matching parameters, the gradient magnitude change at the edge of the optimized super-resolution image patch and the initial key region image patch is calculated; based on the gradient magnitude change, the degree of edge sharpness change is quantified by multi-band fusion to obtain the edge sharpness improvement index. Based on the edge sharpness enhancement index and detail preservation evaluation results, the optimization super-resolution image patch was verified to see whether it simultaneously achieved detail enhancement and artifact suppression in key areas, so as to obtain the comprehensive quality verification results. Based on the comprehensive quality verification results, the optimized super-resolution image patch is sharpened to obtain an enhanced image patch. The enhanced image patch and the initial key region image patch are then compared using a full reference process in terms of texture detail and edge sharpness. After the comparison is successful, the super-resolution image patch is output.

[0039] Furthermore, this invention constructs a full-chain verification system from spatial registration and texture matching to gradient quantization analysis, and integrates multi-dimensional indicators to achieve accurate comparison and controllable enhancement of super-resolution effect and original image details.

[0040] In this embodiment of the invention, low-light and blurred regions are automatically identified by calculating the average gradient and brightness values ​​of the image regions. The key regions include faces, text, and the central region. A region mask is output for each frame, marking the regions requiring enhancement and super-resolution. AI enhancement processing is then performed. For low-light and blurred regions, GAN-based models (such as Zero-DCE or STFAN) are used for denoising and enhancement. For key regions, they are cropped from the mask and super-resolution is performed using a HAT model at a factor of 4 to 8. The enhanced blocks are then seamlessly fused back into the original image.

[0041] like Figure 6 As shown, this embodiment of the invention also provides an ultra-high-definition video production system based on the fusion of 5G and VR, the system comprising: The video acquisition and transmission module 100 is used to acquire VR panoramic video based on a VR camera array and upload the original VR panoramic video stream to cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during the transmission process. The multi-view alignment module 200 is used to align multi-lens images using feature point matching and optical flow methods, automatically identify and repair stitching seams, and generate seamless panoramic video. The image quality enhancement and reconstruction module 300 is used to enhance the image quality of low-light or blurred areas and perform local super-resolution reconstruction of key areas; it generates multiple low-resolution versions through convolutional downsampling to adapt to the performance of different terminals. The immersive video review module 400 is used for immersive video review based on VR headsets, loading virtual review space, identifying quality problems in panoramic videos, and displaying visual labels in 3D space; The film review report generation module 500 is used to annotate films in a virtual film review space using a VR controller to obtain an interactive film review report.

[0042] In a preferred embodiment of the present invention, the video acquisition and transmission module 100 includes: The video slicing encoding unit is used to cut the original VR panoramic video stream into small data packets at predetermined time intervals, perform FEC encoding, generate original data packets, and calculate redundancy check packets. The network detection unit is used to continuously send detection packets to the camera, calculate round-trip time, packet loss rate and jitter, and obtain the network status. The adaptive bitrate control unit is used to dynamically adjust the output bitrate of the video encoder based on the network status feedback from the probe, and to dynamically adjust the ratio of the number of raw data packets to the number of redundant check packets.

[0043] In a preferred embodiment of the present invention, the multi-screen alignment module 200 includes: The feature point matching unit is used to extract several feature points that are not affected by lighting and angle based on the overlapping area images of adjacent shots; to perform preliminary matching on the feature points, find the corresponding pixel points in the two images, and obtain feature point matching pairs; The motion vector calculation unit is used to calculate the motion vector of each pixel in the overlapping region based on the deep learning optical flow algorithm, and to determine the optical flow information. The image alignment unit is used to retrieve the physical calibration parameters of the VR camera array and perform image alignment based on the physical calibration parameters, feature point matching pairs, and optical flow information.

[0044] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0045] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0046] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0047] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for producing ultra-high-definition video based on the integration of 5G and VR, characterized in that, The method includes the following steps: VR panoramic video is captured using a VR camera array and uploaded to the cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during transmission. Feature point matching and optical flow are used to align multi-lens images, automatically identify and repair stitching seams, and generate seamless panoramic videos. Image quality is enhanced in low-light or blurry areas, and local super-resolution reconstruction is performed in key areas; multiple low-resolution versions are generated through convolutional downsampling to adapt to different terminal performance. Immersive film review based on VR headset, loading virtual review space, identifying quality problems in panoramic videos, and displaying visual labels in 3D space; Annotate films in a virtual review space using a VR controller to obtain an interactive review report; The steps of enhancing image quality in low-light or blurred areas and performing local super-resolution reconstruction in key areas specifically include: A semantic segmentation network is used to analyze panoramic frames, identify low-light areas, blurred areas and key areas, and output a region mask for each frame to mark the areas that need to be enhanced and super-resolved. For low-light and blurred regions, a GAN-based model is used for denoising and enhancement; for critical regions, they are cropped from the mask and super-resolution is performed using a HAT model. Specifically, for key regions, the data is cropped from the mask and super-resolution is performed using the HAT model. The specific steps are as follows: A semantic segmentation network is used to process the region mask to extract the boundary coordinates and pixel range information of key regions and obtain the precise location data of the key regions; the key regions include faces, text, and the central region; Based on the precise location data of the key regions, the corresponding key region image patches that retain the original resolution and color space are cropped from the panoramic frame to obtain the initial key region image patches. Load the pre-trained HAT model to obtain a HAT model instance for super-resolution of key regions; The initial key region image patch is input into the HAT model instance for key region super-resolution and super-resolution processing is performed. At the same time, the texture details and edge sharpness of the image patch are enhanced through the attention mechanism to obtain high-resolution key region image patches. Local contrast adjustment and artifact suppression are performed on high-resolution key region image patches to obtain optimized super-resolution image patches; By using a full-reference image quality assessment method, the detail enhancement effect of the optimized super-resolution image patch is compared with that of the initial key region image patch, so as to complete the super-resolution and obtain the super-resolution image patch. Specifically, for key regions, local contrast adjustment and artifact suppression processing are performed on high-resolution key region image blocks to obtain optimized super-resolution image blocks. The specific steps are as follows: The key area is divided into local pixel blocks of a preset size, and the average brightness value and average gradient value of each local pixel block are calculated; the average gradient value is obtained by statistically analyzing the change in pixel intensity. A first threshold is determined based on the average brightness value of local pixel blocks, and a second threshold is determined based on the average gradient value of local pixel blocks. Local pixel blocks with average brightness values ​​lower than the first threshold and average gradient values ​​lower than the second threshold are classified as low-contrast regions. Gray-scale transformation is applied according to the semantic category to which the low-contrast region belongs to obtain contrast-enhanced image blocks. Specifically, the gray-scale transformation is as follows: for face regions, an S-curve is used for processing; for text regions, linear stretching is used; and for the central region, gamma correction is used. Texture features are extracted from contrast-enhanced image patches using a semantic segmentation network; artifact regions are identified by edge and texture feature matching using contrast-enhanced image patches and texture features to obtain artifact location mapping. Weak filtering and smoothing are performed on the gradient intensity of the artifact region determined by the artifact location mapping, and a transition band is constructed at the boundary between the artifact region and the normal region to eliminate local inconsistencies, so as to obtain the artifact-suppressed image patch. The artifact-suppressed image patch and the high-resolution key region image patch are used as reference image quality assessments to obtain detail preservation assessment results; based on the detail preservation assessment results, the artifact-suppressed image patch is sharpened to obtain a fine-tuned image patch; The detail enhancement effect of the fine-tuned image patch was compared with that of the high-resolution key region image patch to verify the artifact suppression results and obtain the optimized super-resolution image patch.

2. The ultra-high-definition video production method based on 5G and VR fusion according to claim 1, characterized in that, The steps involving forward error correction and adaptive rate control specifically include: The original VR panoramic video stream is cut into small data packets at predetermined time intervals, FEC encoded, the original data packets are generated, and redundancy check packets are calculated. Continuously send probe packets to the camera, calculate round-trip time, packet loss rate and jitter, and obtain the network status; Based on the network status reported by the probe, the output bitrate of the video encoder is dynamically adjusted, and the ratio of the number of original data packets to the number of redundant check packets is also dynamically adjusted.

3. The ultra-high-definition video production method based on 5G and VR fusion according to claim 2, characterized in that, The steps for aligning multi-lens images using feature point matching and optical flow specifically include: Based on the overlapping area images of adjacent shots, several feature points that are not affected by lighting and angle are extracted; the feature points are initially matched to find corresponding pixels in the two images, thus obtaining feature point matching pairs; Based on a deep learning optical flow algorithm, the motion vector of each pixel in the overlapping region is calculated to determine the optical flow information; The physical calibration parameters of the VR camera array are retrieved, and the image is aligned based on the physical calibration parameters, feature point matching pairs, and optical flow information.

4. The ultra-high-definition video production method based on 5G and VR fusion according to claim 1, characterized in that, For critical regions, a full-reference image quality assessment method is used to compare the detail enhancement effects of the optimized super-resolution image patch with the initial critical region image patch, in order to complete super-resolution and obtain the super-resolution image patch. The specific steps are as follows: The panoramic frame is processed by a semantic segmentation network to obtain the boundary coordinates of the key region. Using the boundary coordinates of the key region, the optimized super-resolution image patch and the initial key region image patch are spatially registered in the pixel coordinate system to ensure that the optimized super-resolution image patch and the initial key region image patch completely overlap in the pixel range of the corresponding face, text and center region, so as to obtain spatially aligned image pairs. Extract optimized super-resolution image patch texture feature vectors and initial key region image patch texture feature vectors from spatially aligned image pairs; perform point-by-point comparisons between the super-resolution image patch texture feature vectors and the initial key region image patch texture feature vectors to obtain texture matching degree parameters; Using the texture matching parameter, the gradient magnitude change at the edge between the optimized super-resolution image patch and the initial key region image patch is calculated; Based on the gradient amplitude change, the degree of edge sharpness change is quantified by multi-band fusion to obtain the edge sharpness improvement index; Based on the edge sharpness enhancement index and detail preservation evaluation results, the optimization super-resolution image patch was verified to see whether it simultaneously achieved detail enhancement and artifact suppression in key areas, so as to obtain the comprehensive quality verification results. Based on the comprehensive quality verification results, the optimized super-resolution image patch is sharpened to obtain an enhanced image patch. The enhanced image patch and the initial key region image patch are then compared using a full reference process in terms of texture detail and edge sharpness. After the comparison is successful, the super-resolution image patch is output.

5. The ultra-high-definition video production method based on 5G and VR fusion according to claim 4, characterized in that, When automatically identifying and repairing seams, in overlapping areas, graph cut or dynamic programming algorithms are used to determine the path with the lowest energy as the seam; multi-band fusion is performed to decompose the image into different frequency bands, wide-range feathering is performed in the low-frequency band to eliminate exposure differences, and sharp stitching is performed in the high-frequency band to preserve details.

6. The ultra-high-definition video production method based on 5G and VR fusion according to claim 5, characterized in that, Multi-band fusion is performed, decomposing the image into different frequency bands. Wide-range feathering is applied to the low-frequency band to eliminate exposure differences, while sharp stitching is performed to preserve details in the high-frequency band. The specific steps are as follows: The Laplacian pyramid fusion method is used to decompose panoramic images into a predetermined number of frequency bands to obtain low-frequency bands and high-frequency bands; where smooth areas of the image represent low-frequency bands and image edges and textured areas represent high-frequency bands. Wide-range feathering is applied to the decomposed low-frequency band: by smoothing the brightness gradient within the low-frequency band, the exposure differences between different lenses are eliminated, and the feathered low-frequency band is obtained. By using the low-frequency band after feathering, the exposure of the overlapping area is balanced by adjusting the feathering range and intensity, so as to eliminate the brightness difference caused by different exposure of different lenses and obtain a low-frequency band with consistent exposure. Based on the low-frequency bands with consistent exposure, the decomposed high-frequency bands are sharply stitched together to obtain the stitched high-frequency bands. Using the pixel color difference and texture feature difference in the overlapping area of ​​adjacent lenses as energy terms, a graph cut algorithm is used to optimize the stitching path of the stitched high-frequency bands: the stitching path along the lowest energy term is extended to avoid face areas and edge areas, and the texture details of the image corresponding to the stitched high-frequency bands are preserved by minimizing the gradient difference of edge pixels between adjacent lenses, so as to obtain high-frequency bands with enhanced details. A weighted average is used to fuse the low-frequency bands with consistent exposure with the high-frequency bands with enhanced details to obtain a preliminary fused panoramic image. The preliminary fused panoramic image is then subjected to overall smoothing adjustments to obtain a seamless panoramic video.

7. An ultra-high-definition video production system based on the integration of 5G and VR, characterized in that, The system employs the ultra-high-definition video production method based on 5G and VR fusion as described in any one of claims 1 to 6 above, and the system comprises: The video acquisition and transmission module is used to acquire VR panoramic video based on a VR camera array and upload the original VR panoramic video stream to cloud storage in real time via a 5G network; forward error correction and adaptive bitrate control are used during the transmission process. The multi-view alignment module is used to align multi-lens images using feature point matching and optical flow methods, automatically identify and repair stitching seams, and generate seamless panoramic videos. The image quality enhancement and reconstruction module is used to enhance the image quality of low-light or blurred areas and perform local super-resolution reconstruction of key areas; it generates multiple low-resolution versions through convolutional downsampling to adapt to the performance of different terminals. The immersive review module is used for immersive review based on VR headsets, loading a virtual review space, identifying quality issues in panoramic videos, and displaying visual labels in 3D space; The film review report generation module is used to annotate films in a virtual film review space using a VR controller, and generate interactive film review reports.

8. The ultra-high-definition video production system based on 5G and VR fusion according to claim 7, characterized in that, The video acquisition and transmission module includes: The video slicing encoding unit is used to cut the original VR panoramic video stream into small data packets at predetermined time intervals, perform FEC encoding, generate original data packets, and calculate redundancy check packets. The network detection unit is used to continuously send detection packets to the camera, calculate round-trip time, packet loss rate and jitter, and obtain the network status. The adaptive bitrate control unit is used to dynamically adjust the output bitrate of the video encoder based on the network status fed back by the probe, and to dynamically adjust the ratio of the number of original data packets to the number of redundant check packets. The multi-screen alignment module includes: The feature point matching unit is used to extract several feature points that are not affected by lighting and angle based on the overlapping area images of adjacent shots; to perform preliminary matching on the feature points, find the corresponding pixel points in the two images, and obtain feature point matching pairs; The motion vector calculation unit is used to calculate the motion vector of each pixel in the overlapping region based on the deep learning optical flow algorithm, and to determine the optical flow information. The image alignment unit is used to retrieve the physical calibration parameters of the VR camera array and perform image alignment based on the physical calibration parameters, feature point matching pairs, and optical flow information.

Images