High-resolution video transmission method and system based on color layering and frame extraction

Abstract

The invention provides a high-resolution video transmission method and system based on color layering and frame extraction, and relates to the technical field of video compression transmission. The method provided by the invention comprises the following steps: converting an input high-resolution color video into a high-resolution black-and-white video, and dynamically adjusting a sampling rate according to a real-time broadband to generate a low-pixel black-and-white video; acquiring a full-color key frame by adopting a mixed frame extraction mechanism; and during playback, automatically matching the full-color key frame with the nearest time as a rendering anchor point, and performing up-sampling on the low-pixel black-and-white video and rendering the low-pixel black-and-white video into a high-resolution full-color video through an improved Reinhard color migration algorithm. According to the method, high-efficiency transmission and restoration of a high-resolution video are realized by combining color layering, a dynamic frame extraction mechanism and an edge constraint layering color migration algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video compression transmission, and particularly relates to a high-resolution video transmission method and system based on color layering and frame extraction. BACKGROUND

[0002] In the field of video transmission, with the wide popularity of 4K / 8K ultra-high-definition videos, traditional video transmission technologies are facing severe challenges. According to the report released by China Electronics Standardization Institute in 2022, the original code rate of 4K video without compression is as high as about 80 Mbps, which has far exceeded the actual carrying capacity of the current mainstream wireless network - the average downlink rate of 4G network is only 25-50 Mbps, and even in the high-load scenario of 5G network, the actual transmission rate is usually in the range of 30-60 Mbps.

[0003] The existing video output technology has the following defects: the compression efficiency of the traditional video coding standard is limited in a low-bandwidth environment, the picture quality decreases significantly, especially the color detail part is seriously lost; the existing frame extraction strategy is usually single and fixed, and cannot be dynamically adjusted according to the picture content, resulting in the loss of key frames with motion changes or rich colors; the adaptive code rate switching does not respond smoothly in the real network environment, and the frequent clarity fluctuations affect the viewing experience; there is no hierarchical processing mechanism for color information, and the visual characteristics are not fully utilized to optimize the compression efficiency; the general color migration algorithm is prone to edge bleeding, texture blur and other picture quality problems when processing complex scenes. Therefore, it is urgent to provide a solution to improve the above problems. SUMMARY

[0004] The purpose of the present application is to provide a high-resolution video transmission method and system based on color layering and frame extraction, which can improve the color restoration distortion, high redundancy of frame extraction and large rendering delay in the existing video transmission technology.

[0005] In a first aspect, the present application provides a high-resolution video transmission method based on color layering and frame extraction, comprising:

[0006] The high-resolution color video is subjected to color space conversion to obtain a high-resolution black-and-white video; the high-resolution black-and-white video is subjected to edge enhancement processing based on a Laplacian operator; the processed full-pixel black-and-white video is converted into a low-pixel black-and-white video based on real-time bandwidth dynamic adjustment of the downsampling rate;

[0007] The multi-scale structural similarity index between consecutive frames in the high-resolution color video is calculated, if the multi-scale structural similarity index is lower than a first preset threshold, a frame extraction operation is triggered to obtain a full-color key frame, if the multi-scale structural similarity index is not lower than the first preset threshold, the Bhattacharyya distance of the RGB channel histogram of consecutive frames in the high-resolution video is calculated, if the Bhattacharyya distance is higher than a second preset threshold, the frame extraction operation is triggered to obtain the full-color key frame, if the Bhattacharyya distance is not higher than the second preset threshold, whether the frame extraction operation interval time exceeds a preset time length is judged, if yes, the frame extraction operation is triggered to obtain the full-color key frame, and if no, the frame extraction operation is not triggered.

[0008] The low-pixel black-and-white video is restored to a high-resolution black-and-white video based on an image upsampling method, adaptive Canny edge detection is performed on the high-resolution black-and-white video and the full-color key frame respectively to generate corresponding black-and-white frame edge masks and key frame edge masks, and the black-and-white frame edge masks and the key frame edge masks are fused to generate a joint edge mask; the color of the full-color key frame is migrated to the high-resolution black-and-white video based on an improved Reinhard color migration algorithm and the joint edge mask to obtain a restored high-resolution full-color video.

[0009] The application provides a high-resolution video transmission method based on color layering and frame extraction, which reduces the bandwidth through two-stage color conversion of high-resolution color video, ensures that key information is not lost by combining variable triggering frame extraction, and realizes high-quality restoration by using the latest full-color frame extraction as a rendering anchor point.

[0010] Optionally, when the high-resolution color video is subjected to color space conversion to obtain a high-resolution black-and-white video, the high-resolution color video is subjected to YUV color space conversion, only the luminance channel of the high-resolution color video is retained, and the chrominance channel is discarded to obtain the high-resolution black-and-white video.

[0011] Optionally, the image upsampling method comprises bicubic interpolation and edge-directed interpolation.

[0012] Optionally, when the full-color key frame is subjected to adaptive Canny edge detection to generate a key frame edge mask, a clipping threshold is determined based on a cumulative distribution function, the chrominance channel of the full-color key frame is preprocessed based on the clipping threshold, the preprocessed full-color key frame is converted into a YUV color space and the luminance channel is extracted, and the key frame edge mask is generated based on adaptive Canny edge detection.

[0013] Optionally, when the black-and-white frame edge mask and the key frame edge mask are fused to generate a joint edge mask, a morphological closing operation or a weighted fusion method is adopted to fuse the black-and-white frame edge mask and the key frame edge mask to generate the joint edge mask.

[0014] Optionally, when the color of the full-color key frame is migrated to the high-resolution black-and-white video based on the improved Reinhard color migration algorithm and the joint edge mask to obtain the restored high-resolution full-color video, the method comprises: respectively calculating statistical values of a luminance channel of the high-resolution black-and-white video and statistical values of a chrominance channel of the full-color key frame, wherein the statistical values of the luminance channel of the high-resolution black-and-white video are obtained by weighting based on the joint edge mask; determining different weight factors based on the joint edge mask, and constructing a color migration formula, based on which the color of the full-color key frame is migrated to the high-resolution black-and-white video to obtain the restored high-resolution full-color video.

[0015] Optionally, the color migration formula is as follows:

[0016] ,

[0017] wherein, represents the chrominance value of the output image at position ; represents the original chrominance value of the input image at position ; and respectively represent the weighted mean value and the standard deviation of the luminance channel of the high-resolution black-and-white video; and respectively represent the mean value and the standard deviation of the corresponding chrominance channel of the full-color key frame; represents the weight factor, which is defined by a piecewise function as , represents the joint edge mask, represents an edge confidence threshold; represents a local edge alignment term weighted by a Gaussian kernel.

[0018] In a second aspect, the present application provides a high-resolution video transmission system based on color layering and frame extraction, comprising:

[0019] a color layering module, configured to perform color space conversion on a high-resolution color video to obtain a high-resolution black-and-white video, perform edge enhancement processing on the high-resolution black-and-white video based on a Laplacian operator, and convert the processed full-pixel black-and-white video into a low-pixel black-and-white video based on real-time bandwidth dynamic adjustment of a downsampling rate;

[0020] The frame extraction module is configured to calculate a multi-scale structural similarity index between consecutive frames in the high-resolution video, and if the multi-scale structural similarity index is lower than a first preset threshold, a frame extraction operation is triggered to obtain a full-color key frame; if the multi-scale structural similarity index is not lower than the first preset threshold, a Bhattacharyya distance of RGB channel histograms between consecutive frames in the high-resolution video is calculated, and if the Bhattacharyya distance is higher than a second preset threshold, the frame extraction operation is triggered to obtain the full-color key frame; if the Bhattacharyya distance is not higher than the second preset threshold, whether the frame extraction operation interval time exceeds a preset time length is determined, and if yes, the frame extraction operation is triggered to obtain the full-color key frame, and if no, the frame extraction operation is not triggered.

[0021] The color migration module is configured to restore the low-pixel black-and-white video to a high-resolution black-and-white video based on an image upsampling method, perform adaptive Canny edge detection on the high-resolution black-and-white video and the full-color key frame respectively to generate corresponding black-and-white frame edge masks and key frame edge masks, and fuse the black-and-white frame edge masks and the key frame edge masks to generate a joint edge mask; and based on an improved Reinhard color migration algorithm and the joint edge mask, the color of the full-color key frame is migrated to the high-resolution black-and-white video to obtain a restored high-resolution full-color video. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a high-resolution video transmission method based on color layering and frame extraction provided by an embodiment of the present application is shown in FIG. 1.

[0023] Figure 2 A structural diagram of a high-resolution video transmission system based on color layering and frame extraction provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should have the meanings commonly understood by those skilled in the art with ordinary skills.

[0025] Referring to Figure 1 The present application provides a high-resolution video transmission method based on color layering and frame extraction, which comprises the following steps:

[0026] S1, color space conversion is performed on the high-resolution color video to obtain a high-resolution black-and-white video; edge enhancement is performed on the high-resolution black-and-white video based on a Laplacian operator; and the processed full-pixel black-and-white video is converted into a low-pixel black-and-white video based on real-time bandwidth dynamic adjustment of a downsampling rate;

[0027] S2, a multi-scale structural similarity index between consecutive frames in the high-resolution color video is calculated, if the multi-scale structural similarity index is lower than a first preset threshold, a frame extraction operation is triggered to obtain a full-color key frame, if the multi-scale structural similarity index is not lower than the first preset threshold, a Bhattacharyya distance of RGB channel histograms between consecutive frames in the high-resolution video is calculated, if the Bhattacharyya distance is higher than a second preset threshold, the frame extraction operation is triggered to obtain the full-color key frame, if the Bhattacharyya distance is not higher than the second preset threshold, whether an interval time of the frame extraction operation exceeds a preset time length is judged, if yes, the frame extraction operation is triggered to obtain the full-color key frame, and if no, the frame extraction operation is not triggered;

[0028] S3, the low-pixel black-and-white video is restored to a high-resolution black-and-white video based on an image upsampling method, adaptive Canny edge detection is respectively performed on the high-resolution black-and-white video and the full-color key frame to generate corresponding black-and-white frame edge masks and key frame edge masks, and the black-and-white frame edge masks and the key frame edge masks are fused to generate a joint edge mask; color of the full-color key frame is migrated to the high-resolution black-and-white video based on an improved Reinhard color migration algorithm and the joint edge mask, and a restored high-resolution full-color video is obtained.

[0029] In fact, the high-resolution video transmission method provided by the application converts the input high-resolution color video into a high-resolution black-and-white video, dynamically adjusts the sampling rate according to the real-time bandwidth, generates a low-pixel black-and-white video, uses a hybrid frame extraction mechanism to obtain a full-color key frame, automatically matches the time closest full-color key frame as a rendering anchor point during playback, and uses the improved Reinhard color migration algorithm, so that the low-pixel black-and-white video can be upsampled and rendered into a high-resolution full-color video, realizing efficient transmission and high-quality restoration of the high-resolution video.

[0030] In some embodiments, when the high-resolution color video is converted into a high-resolution black-and-white video in step S1, the input high-resolution color video is converted from the RGB color space to the YUV color space based on YUV color space conversion, only the Y luminance channel is retained, and the U and V chrominance channels are discarded to obtain a high-resolution black-and-white video. The Y luminance channel carries the light and dark changes, contour structure and detail information of the image, i.e., the black-and-white picture content; retaining the Y channel can greatly save the transmission bandwidth. After this step, the video is converted into a pure black-and-white picture, which lays a foundation for subsequent high-quality rendering while retaining the key structure, scene content and motion information. The U and V components correspond to color information, i.e., chrominance. The luminance channel calculation formula is as follows:

[0031] ,

[0032] wherein, represents the red component intensity value (usually 0-255) of the input pixel, represents the green component intensity value (usually 0-255) of the input pixel, represents the blue component intensity value (usually 0-255) of the input pixel.

[0033] In some embodiments, when the high-resolution black-and-white video is edge-enhanced based on the Laplacian operator in step S1, the edge is enhanced based on the Laplacian operator before the high-resolution black-and-white video is down-sampled, so that the contour of the video is clearer and avoids blurring, thereby maintaining the recognizability of the video picture as much as possible at a very low code rate.

[0034] In some embodiments, when the processed full-pixel black-and-white video is converted into a low-pixel black-and-white video based on real-time bandwidth dynamic adjustment of the down-sampling rate in step S1, the system continuously detects the network condition with a period of 500 ms and dynamically adjusts the down-sampling ratio accordingly. Through the processing procedure of first converting into a high-resolution black-and-white video and then down-sampling, the amount of video stream data generated is significantly reduced, so that stable transmission can be achieved under poor network conditions. The specific adjustment strategy is as follows:

[0035] If the real-time bandwidth Bw is greater than or equal to 8 Mbps, the down-sampling rate is 1 / 2;

[0036] If the real-time bandwidth 4 Mbps≤Bw<8Mbps, the down-sampling rate is 1 / 4;

[0037] If the real-time bandwidth Bw is less than 4 Mbps, the down-sampling rate is 1 / 8.

[0038] For better illustration, taking a 4K@30fps video (original code rate 80 Mbps) as an example, the processing effect under different bandwidth scenarios is as follows:

[0039] High bandwidth scenario (Bw=10Mbps): downsample to 1080P, transmission bandwidth 4.8Mbps;

[0040] Medium bandwidth scenario (Bw=5Mbps): downsample to 540P, transmission bandwidth 2.1Mbps;

[0041] Low bandwidth scenario (Bw=2Mbps): downsample to 360P, transmission bandwidth 1.0Mbps.

[0042] In some embodiments, in step S2, the multi-scale structural similarity index between consecutive frames in the high-resolution color video is calculated, and if the multi-scale structural similarity index is lower than a first preset threshold, the multi-scale structural similarity (MS-SSIM) detection is used to objectively measure the perceptual similarity between two images (usually one original reference image and one distorted image). The SSIM value of the high-resolution color video between consecutive frames at three different scales is calculated, and if it is lower than the first preset threshold (which can be set to 0.85), it is determined that the picture is changing, and the frame extraction operation is triggered to obtain the full-color key frame. The MS-SSIM value of the high-resolution color video between consecutive frames at three different scales is calculated, and if it is lower than the first preset threshold (which can be set to 0.85), it is determined that the picture is changing, and the frame extraction operation is triggered to obtain the full-color key frame. The MS-SSIM value of the high-resolution color video between consecutive frames at three different scales is calculated, and if it is lower than the first preset threshold (which can be set to 0.85), it is determined that the picture is changing, and the frame extraction operation is triggered to obtain the full-color key frame. The formula used is as follows:

[0043] ,

[0044] wherein, represents the luminance comparison function, represents the number of scales, represents the contrast and structure comparison function, and represent the weight coefficients.

[0045] Specifically, the first preset threshold is automatically adjusted according to the scene texture complexity, so that the sensitivity is reduced in the texture complex area (such as leaves, fabrics), to avoid false triggering, and the sensitivity is improved in the smooth area (such as walls, sky), to ensure that small changes can be captured. The specific steps are as follows:

[0046] S21, input a high-resolution black and white video frame, set the current frame as , and the size is H*W;

[0047] S22, calculate the image gradient in the horizontal direction and the vertical direction based on the Sobel operator, and the formula used is as follows:

[0048] ,

[0049] ,

[0050] wherein, and Representing images respectively Sobel convolution kernels in the horizontal direction Gradient map and image obtained after convolution operation Sobel convolution kernels in the vertical direction The gradient map obtained after performing convolution operations;

[0051] S23. Calculate the gradient magnitude using the following formula:

[0052] ,

[0053] in, Representing coordinates gradient magnitude at that point and Representing coordinates The horizontal and vertical gradients at the location;

[0054] S24. Calculate the variance of the gradient magnitude, i.e., the texture complexity, using the following formula:

[0055] ,

[0056] in, This represents the final texture complexity, which can be set to an empirical value (such as 5000) or calculated by using a sliding window to count the historical maximum values. This represents the mean of the gradient magnitude, i.e. ; Indicates the height of the image; Indicates the width of the image;

[0057] S25. Texture complexity can be normalized to the [0,1] interval according to the actual situation, using the following formula:

[0058] ,

[0059] in, This represents the normalized texture complexity, which is convenient for subsequent threshold calculation; This represents the maximum value of the texture complexity.

[0060] S26. The system automatically adjusts the first preset threshold based on the scene texture complexity: if the texture is complex and detailed (e.g., complexity reaches 0.9), the threshold is automatically lowered to 0.76 to suppress false alarms caused by texture fluctuations; if the texture is smooth and simple (e.g., complexity is 0.1), the threshold is automatically raised to 0.84, thus becoming more sensitive to subtle real-world changes and avoiding missed detections. The formula used is as follows:

[0061] ,

[0062] wherein, represents the first preset threshold value adjusted dynamically.

[0063] In some embodiments, in step S2, the Bhattacharyya distance of the RGB channel histogram of consecutive frames in the high-resolution video is calculated, and if the Bhattacharyya distance is higher than the second preset threshold value, the frame extraction operation is triggered to obtain the full-color key frame. When the Bhattacharyya distance of the RGB channel histogram of consecutive frames in the high-resolution video is calculated, if the Bhattacharyya distance is higher than the second preset threshold value (which can be set to 0.15), the system determines that the video picture has changed in light or color, and triggers the frame extraction operation to extract the full-color key frame. The RGB histogram decomposes the image color information into three channels of red (R), green (G), and blue (B). The histogram of each channel reflects the number of pixels of the color at each intensity value from 0 (darkest) to 255 (brightest), which is used to describe the overall color distribution of the image, but does not contain any spatial position information. By comparing the shapes of the R, G, and B histograms of two consecutive frames, the similarity of their color distributions can be objectively evaluated: if the two frames are visually close in color, their histogram shapes are also similar; otherwise, it indicates that there may be overall color change, brightness change, or contrast change. The calculation formula used is as follows:

[0064] ,

[0065] wherein, represents the histogram difference value, i.e., the Bhattacharyya distance; and represent two normalized histograms.

[0066] In some embodiments, in step S2, to prevent the aging of anchor frames caused by long static scenes, if the Bhattacharyya distance is not higher than the second preset threshold value, it is determined whether the frame extraction operation interval time exceeds the preset time length (which can be set to 1 second), and if the frame extraction operation interval time exceeds the preset time length, the frame extraction operation is triggered to obtain the full-color key frame, and if the frame extraction operation interval time does not exceed the preset time length, the frame extraction operation is not triggered.

[0067] In the test based on the CDNet2014 dataset, the key frame capture strategy of the present embodiment can adaptively adjust according to the scene content: in a motion scene, the average frame extraction interval is 3.2 seconds; in a static scene, the frame extraction interval is stabilized at 1.0 seconds triggered by the bottom protection mechanism; when the scene has a sudden change in light, the system can respond instantly and capture the key frame immediately when the light compensation is turned on.

[0068] In some embodiments, in step S3, when the low-pixel black-and-white video is restored to a high-resolution black-and-white video based on the image up-sampling method, the low-pixel black-and-white video (such as 540P) is up-sampled to the original high-resolution black-and-white video (such as 4K) by using bicubic interpolation or edge-directed interpolation (EDI), so as to provide a spatially aligned high-resolution highlight map for subsequent color migration, and to avoid color errors caused by resolution mismatch. In addition, in the present embodiment, edge-directed interpolation is preferred. This algorithm guides the interpolation process by the local gradient direction, and realizes resolution enhancement while preserving sharp edges.

[0069] In some embodiments, in step S3, when the adaptive Canny edge detection is performed on the high-resolution black-and-white video to generate a black-and-white frame edge mask, the up-sampled high-resolution black-and-white frame is subjected to adaptive Canny detection to dynamically calculate high and low thresholds, and generate a binary edge mask representing the structure of the current frame , which is used to represent the contour region and is used for subsequent color migration weighting to prevent excessive color diffusion in non-edge regions.

[0070] Specifically, the high and low thresholds are dynamically set according to the local contrast of the image, and the formula is as follows:

[0071]

[0072]

[0073] wherein, represents the low threshold of the Canny operator; represents the high threshold of the Canny operator; represents the low threshold proportion coefficient, and is commonly taken as ; represents the low threshold proportion coefficient, and is commonly taken as ; represents the global contrast estimation of the image; represents the high-resolution black-and-white frame.

[0074] In some embodiments, in step S3, when the adaptive Canny edge detection is performed on the full-color key frame to generate a key frame edge mask, histogram analysis is performed on the full-color key frame to determine 5% and 95% clipping thresholds, and the extreme pixel values are clipped, and the preprocessed full-color key frame is converted to the YUV color space, the Y luminance channel is extracted, and the same edge detection is performed to obtain the key frame edge mask ​​Its function is to ensure that color migration is carried out with high weight only in structurally consistent edge regions, avoiding incorrect coloring of edges that do not exist in the keyframe.

[0075] In some embodiments, when fusing the black-and-white frame edge mask and the keyframe edge mask to generate a joint edge mask in step S3, the black-and-white frame edge mask and the keyframe edge mask are fused using morphological closing operations or weighted fusion methods to generate the joint edge mask. Its function is to impart high confidence in the overlapping areas of the two edges, reducing edge inconsistencies caused by motion or lighting. In this embodiment, morphological closing operations are used to smooth the edges.

[0076] Specifically, the formula used for morphological closing operations is as follows:

[0077] ; in, This represents the joint edge mask of the output; This indicates a morphological expansion operation, used to expand areas, fill holes, and connect adjacent areas. This indicates a morphological etching operation used to reduce the area, eliminate minor noise, and separate adjacent areas. Represents the morphological closing operation;

[0078] Specifically, the formula used for weighted fusion is as follows:

[0079] ,

[0080] ,

[0081] in, and These represent the edge mask weights of black and white frames and the edge mask weights of keyframes, respectively. This represents the joint edge mask obtained after weighted fusion; Represents the binary edge mask of the current frame; This represents the edge mask of the keyframe.

[0082] In some embodiments, in step S3, based on the improved Reinhard color migration algorithm and joint edge mask, the colors of the full-color keyframes are migrated to the high-resolution black-and-white video to obtain the restored high-resolution full-color video. Then, the statistical values ​​(mean and variance statistics) of the luminance channel of the high-resolution black-and-white video and the statistical values ​​(mean and variance statistics) of the chroma channel of the full-color keyframes are calculated respectively. The statistical values ​​of the luminance channel of the high-resolution black-and-white video are obtained by weighting using the joint edge mask. An edge weight factor is introduced based on the standard color migration formula. This weighting factor Determined by the joint edge mask: edge region ( , strengthen color transfer, maintain structural consistency; non-edge region ( , weaken color influence, suppress noise diffusion. A color transfer formula is constructed in this way, and the color of the full-color key frame is transferred to the high-resolution black-and-white video based on the color transfer formula to obtain the restored high-resolution full-color video. In addition, a local alignment term weighted by Gauss can be combined to ensure that the color transition is natural and smooth. The color transfer formula is as follows:

[0083]

[0084] wherein, represents the chroma channel of the output image at position ; represents the original chroma value of the input image at position ; and respectively represent the weighted mean and standard deviation of the luminance channel of the high-resolution black-and-white video; and respectively represent the mean and standard deviation of the corresponding chroma channel of the full-color key frame; represents the weight factor, which is defined by a piecewise function as represents the joint edge mask, represents the edge confidence threshold; represents the local edge alignment term weighted by Gauss kernel.

[0085] In addition, GPU parallel computing is used to speed up the statistical quantity operation; the time domain re-projection technology is used to reuse the inter-frame mask to reduce the temporal flicker; and the CIDA Thrust library is used to optimize the histogram-related calculation.

[0086] To sum up, the scheme significantly improves the color rendering quality. By introducing edge constraints, color leakage is effectively avoided, and color overflow at the boundary is prevented (for example, the sky blue will not "overflow" to the edge of the building). In a moving scene, there is a frame displacement between the instant black-and-white frame and the key frame. The algorithm based on edge constraints can also guarantee the high consistency of local color. For low-texture areas, setting a low weight effectively suppresses noise coloring, and the color error (CIED2000) is reduced by 37.8%.

[0087] The test results of the embodiment on the NVIDIA Tesla T4 GPU are as follows:

[0088] The single-frame rendering delay of the 4K video is 28 milliseconds, which meets the real-time processing requirement of 30fps;

[0089] ​​​​The color restoration accuracy of the edge region is 23% higher than that of the traditional method;

[0090] In a low-texture scene, the CIEDE2000 color error is reduced to 5.1, and the reduction is 37.8%.

[0091] In the strict test environment of 4K@30fps monitoring video (bandwidth limit 5Mbps, bit rate 80Mbps), the scheme is significantly better than the HEVC-based video transmission scheme (HEVC) and the machine learning-based video transmission scheme (Meta) in many key performance indicators. The specific comparison is as follows:

[0092] Transmission bandwidth: the scheme only needs 3Mbps, which is much lower than HEVC (20Mbps) and Meta scheme (8Mbps);

[0093] Rendering delay: the scheme is 28ms, which is faster than HEVC (45ms) and Meta scheme (65ms);

[0094] Motion scene PSNR: the scheme is 41.7dB, which is significantly higher than HEVC (34.2dB) and Meta scheme (36.5dB) in picture quality;

[0095] Color restoration error (CIEDE2000): the scheme is only 4.1, which is significantly better than HEVC (8.3) and Meta scheme (6.9) in color fidelity.

[0096] The present application can be seamlessly integrated into the existing video transmission system as a codec plug-in, and the working process is as follows: the system captures the original high-resolution color video through the camera; before encoding, the color video is converted into low-resolution black and white video through preprocessing, which greatly reduces the amount of data to be transmitted; then, the processed black and white video stream is compressed by the standard H.265 encoder, and the encoder is treated as ordinary video without any modification; the compressed code stream is sent to the receiving end through the transmission channel; the receiving end uses the standard H.265 decoder to restore the low-resolution black and white video; after decoding, the black and white video is up-sampled and color reconstructed based on the sparse full-color key frame, and finally the high-resolution color video is output to the display terminal.

[0097] The application can be applied in the deployment of intelligent factory monitoring scenes, and the workflow is as follows: the system collects high-resolution video through a 4K industrial camera; color space conversion and dynamic downsampling preprocessing are performed on the video at the edge gateway end, and low-pixel black and white video streams are output; at the same time, based on a hybrid frame extraction mechanism, the picture content is analyzed in real time, and significant changes in structure or illumination color are detected, and full-color key frames are extracted and stored accordingly; the preprocessed black and white video streams and key frame data are transmitted to the control center server through the 5G network; after receiving the data, the server reconstructs high-quality full-color pictures through anchor point matching and real-time rendering technology; finally, the video signal is output on a 4K monitoring large screen, and is connected to a real-time rendering engine and a defect detection AI module to realize intelligent monitoring and quality analysis of the production process.

[0098] Referring to Figure 2 The application provides a high-resolution video transmission system based on color layering and frame extraction, comprising the following steps:

[0099] The color layering module 100 is used for color space conversion of high-resolution color video to obtain high-resolution black and white video; edge enhancement processing is performed on the high-resolution black and white video based on a Laplacian operator; and the processed full-pixel black and white video is converted into low-pixel black and white video based on real-time bandwidth dynamic adjustment of the downsampling rate;

[0100] The frame extraction module 200 is used for calculating a multi-scale structural similarity index between continuous frames in the high-resolution video; if the multi-scale structural similarity index is lower than a first preset threshold, a frame extraction operation is triggered to obtain a full-color key frame; if the multi-scale structural similarity index is not lower than the first preset threshold, a Bhattacharyya distance of an RGB channel histogram between continuous frames in the high-resolution video is calculated; if the Bhattacharyya distance is higher than a second preset threshold, the frame extraction operation is triggered to obtain the full-color key frame; if the Bhattacharyya distance is not higher than the second preset threshold, whether the frame extraction operation interval time exceeds a preset time length is judged; if yes, the frame extraction operation is triggered to obtain the full-color key frame; and if no, the frame extraction operation is not triggered.

[0101] The color migration module 300 is used for restoring the low-pixel black and white video to high-resolution black and white video based on an image upsampling method, performing adaptive Canny edge detection on the high-resolution black and white video and the full-color key frame respectively to generate corresponding black and white frame edge masks and key frame edge masks, and fusing the black and white frame edge masks and the key frame edge masks to generate a joint edge mask; and the color of the full-color key frame is migrated to the high-resolution black and white video based on an improved Reinhard color migration algorithm and the joint edge mask to obtain a restored high-resolution full-color video.

[0102] While the embodiments of the application have been illustrated and described in detail, it will be readily apparent to those skilled in the art that various modifications and changes can be made to the embodiments without departing from the scope and spirit of the application, as described in the claims. Moreover, the application described is not limited in its application to the details set forth in the description or illustrated in the drawings. The application is capable of other embodiments and of being practiced or carried out in various ways.

Claims

1. A high resolution video transmission method based on color layering and frame abstraction, characterized in that, The method comprises the following steps: color space conversion is performed on the high-resolution color video to obtain a high-resolution black-and-white video; edge enhancement processing is performed on the high-resolution black-and-white video based on a Laplacian operator; the full-pixel black-and-white video after processing is converted into a low-pixel black-and-white video based on real-time bandwidth dynamic adjustment of the downsampling rate; a multi-scale structural similarity index between consecutive frames in the high-resolution color video is calculated, and if the multi-scale structural similarity index is lower than a first preset threshold, a frame extraction operation is triggered to obtain a full-color key frame; if the multi-scale structural similarity index is not lower than the first preset threshold, a Bhattacharyya distance of RGB channel histograms of consecutive frames in the high-resolution video is calculated, and if the Bhattacharyya distance is higher than a second preset threshold, the frame extraction operation is triggered to obtain the full-color key frame; if the Bhattacharyya distance is not higher than the second preset threshold, it is determined whether the interval time of the frame extraction operation exceeds a preset time length, and if yes, the frame extraction operation is triggered to obtain the full-color key frame, and if no, the frame extraction operation is not triggered; the low-pixel black-and-white video is restored to a high-resolution black-and-white video based on an image upsampling method, adaptive Canny edge detection is performed on the high-resolution black-and-white video and the full-color key frame respectively to generate a corresponding black-and-white frame edge mask and a key frame edge mask, and the black-and-white frame edge mask and the key frame edge mask are fused to generate a joint edge mask; the color of the full-color key frame is migrated to the high-resolution black-and-white video based on an improved Reinhard color migration algorithm and the joint edge mask to obtain a restored high-resolution full-color video.

2. The high resolution video transmission method of claim 1, wherein, When the color space conversion is performed on the high-resolution color video to obtain the high-resolution black-and-white video, the YUV color space conversion is performed on the high-resolution color video, only the luminance channel of the high-resolution color video is retained, the chrominance channel is discarded, and the high-resolution black-and-white video is obtained.

3. The high resolution video transmission method of claim 1, wherein, The image upsampling method comprises bicubic interpolation and edge-directed interpolation.

4. The high resolution video transmission method of claim 1, wherein, When the adaptive Canny edge detection is performed on the full-color key frame to generate the key frame edge mask, the clipping threshold is determined based on a cumulative distribution function, the chrominance channel of the full-color key frame is preprocessed based on the clipping threshold, the full-color key frame after preprocessing is converted into a YUV color space and the luminance channel is extracted, and the key frame edge mask is generated based on the adaptive Canny edge detection.

5. The high resolution video transmission method of claim 1, wherein, When the black-and-white frame edge mask and the key frame edge mask are fused to generate the joint edge mask, the morphological closing operation or the weighted fusion method is adopted to fuse the black-and-white frame edge mask and the key frame edge mask to generate the joint edge mask.

6. The high resolution video transmission method of claim 1, wherein, The color of a full-color key frame is migrated to a high-resolution black-and-white video based on an improved Reinhard color migration algorithm and a joint edge mask to obtain a restored high-resolution full-color video, including: respectively calculating statistical values of a luminance channel of the high-resolution black-and-white video and statistical values of a chroma channel of the full-color key frame, wherein the statistical values of the luminance channel of the high-resolution black-and-white video are obtained by weighting the joint edge mask; determining different weight factors based on the joint edge mask, and constructing a color migration formula, based on which the color of the full-color key frame is migrated to the high-resolution black-and-white video to obtain the restored high-resolution full-color video.

7. The high resolution video transmission method of claim 6, wherein, The color migration formula is as follows: ， wherein, represents the chrominance value of the output image at position ; represents the original chrominance value of the input image at position ; and respectively represent the weighted mean and standard deviation of the luminance channel of the high-resolution black-and-white video; and respectively represent the mean and standard deviation of the chrominance channels of the full-color keyframe; represents the weight factor, defined by a piecewise function as , represents the joint edge mask, represents the edge confidence threshold; represents the Gaussian kernel weighted local edge alignment term.

8. A high resolution video transmission system based on color layering and frame extraction, characterized in that, including: a color layering module configured to perform color space conversion on a high-resolution color video to obtain a high-resolution black-and-white video; based on a Laplacian operator, performing edge enhancement processing on the high-resolution black-and-white video; based on real-time bandwidth dynamic adjustment of a downsampling rate, converting the processed full-pixel black-and-white video into a low-pixel black-and-white video; a frame extraction module configured to calculate a multi-scale structural similarity index between consecutive frames in a high-resolution video, and if the multi-scale structural similarity index is lower than a first preset threshold, triggering a frame extraction operation to obtain a full-color key frame; if the multi-scale structural similarity index is not lower than the first preset threshold, calculating the Bhattacharyya distance of the RGB channel histogram between consecutive frames in the high-resolution video, and if the Bhattacharyya distance is higher than a second preset threshold, triggering the frame extraction operation to obtain the full-color key frame; if the Bhattacharyya distance is not higher than the second preset threshold, determining whether the frame extraction operation interval time exceeds a preset time length, and if yes, triggering the frame extraction operation to obtain the full-color key frame, and if no, not triggering the frame extraction operation; a color migration module configured to restore the low-pixel black-and-white video to a high-resolution black-and-white video based on an image upsampling method, perform adaptive Canny edge detection on the high-resolution black-and-white video and the full-color key frame respectively to generate corresponding black-and-white frame edge masks and key frame edge masks, and fuse the black-and-white frame edge masks and the key frame edge masks to generate a joint edge mask; based on the improved Reinhard color migration algorithm and the joint edge mask, migrating the color of the full-color key frame to the high-resolution black-and-white video to obtain the restored high-resolution full-color video.

Images