A video transmission display method, device and system based on super-resolution super-frame algorithm

Abstract

The present application relates to the field of video transmission and display technology, and discloses a video transmission and display method, device and system based on a super-resolution superframe algorithm. The method includes building a spliced ​​large screen and obtaining screen information, performing transmission preprocessing on the original video, dividing the original video into image blocks, performing super-resolution superframe processing on the image blocks and obtaining the video to be displayed. The device and the system correspond to the method. The present application provides data support for the video processing target by obtaining the screen information of the spliced ​​large screen; improves the applicability of the method by transmission preprocessing of the original video; clarifies the display position of the original video on the spliced ​​large screen by dividing the image blocks, and optimizes the display effect; provides a reference for super-resolution superframe reconstruction by detecting the color state of the image block and determining the super-resolution superframe target, and reduces the amount of super-resolution superframe reconstruction calculation; realizes super-resolution superframe reconstruction of the image block to be displayed by the super-resolution superframe algorithm, and optimizes the display effect. Finally, the video transmission and display process is optimized.

Description

Technical Field

[0001] The present application relates to the technical field of video transmission and display, and specifically to a method, device and system for video transmission and display based on a super-resolution super-frame algorithm. Background Art

[0002] As the splicing large-screen display technology becomes more and more popular, its use environment is becoming increasingly complex. For example, when the splicing large screen is used outdoors, it is impossible to provide it with high-bandwidth network transmission; the quality of the original video used for display is provided by the user, and there is no guarantee that the original video can be adapted to the splicing large screen.

[0003] Due to the huge amount of computing power in the video processing process, the sub-screens may not display synchronously during the display of the spliced ​​large screen.

[0004] Chinese patent number CN111489292B discloses a method and device for super-resolution reconstruction of a video stream, the method comprising: dividing a target video frame image in a video stream into a plurality of image blocks, detecting the motion state of the image blocks; counting the proportion of image blocks in the target video frame image that are in a stationary state or move slowly; determining the reconstruction frame rate according to the proportion, and determining each video frame image to be reconstructed according to the reconstruction frame rate; performing super-resolution reconstruction on each video frame image to be reconstructed based on a preset super-resolution network model, and obtaining a super-resolution reconstruction result of each video frame image to be reconstructed; using a preset scaling algorithm to enlarge the video stream frame by frame, and obtaining an enlarged result of each video frame image of the video stream; fusing the super-resolution reconstruction results of each video frame image to be reconstructed and the enlarged results of each video frame image, and obtaining a super-resolution reconstruction result of the video stream. This method realizes super-resolution reconstruction of the video stream.

[0005] In summary, the prior art lacks a technical solution for optimizing the video transmission and display process. Summary of the invention

[0006] The purpose of this application is to provide a video transmission and display method, device and system based on a super-resolution super-frame algorithm to solve the technical problems raised in the above-mentioned background technology.

[0007] To achieve the above objectives, this application discloses the following technical solutions:

[0008] In a first aspect, the present application discloses a video transmission and display method based on a super-resolution super-frame algorithm, the method comprising:

[0009] S1: Build a spliced ​​large screen consisting of M*N identical sub-screens, and obtain the sub-screen resolution, sub-screen frame rate, and sub-screen screen ratio; wherein M is the number of sub-screens in the horizontal direction of the spliced ​​large screen, and N is the number of sub-screens in the vertical direction of the spliced ​​large screen; record screen information, which includes the number of sub-screens in the horizontal direction of the spliced ​​large screen, the number of sub-screens in the vertical direction of the spliced ​​large screen, the sub-screen resolution, the sub-screen frame rate, and the sub-screen screen ratio;

[0010] S2: Obtain the original video and the corresponding resolution and frame rate, and determine whether the original video meets the low-bandwidth network transmission requirement. If yes, transmit the original video and execute step S4; otherwise, reduce the resolution and frame rate of the original video to obtain the first video and execute step S3;

[0011] S3: receiving the first video, performing inverse processing of reducing the resolution and frame rate of the first video, restoring the first video to the original video, transmitting the original video and executing step S4;

[0012] S4: receiving the original video and acquiring the screen information;

[0013] S5: dividing each frame in the original video into a plurality of image blocks based on the screen information, and establishing a corresponding relationship between the image blocks and the sub-screens;

[0014] S6: for any of the image blocks, detecting a color state of the image block, wherein the color state is used to determine whether the image block needs to be reconstructed;

[0015] S7: Determine a super-resolution super-frame target based on the screen information;

[0016] S8: Determine each image block to be reconstructed by using the color state and the super-resolution super-frame target;

[0017] S9: Based on a preset super-resolution super-frame algorithm, super-resolution super-frame reconstruction is performed on each of the image blocks to be reconstructed to obtain an image block to be displayed;

[0018] S10: Obtain a video to be displayed using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

[0019] Through the above, by acquiring the screen information of the spliced ​​large screen, data support is provided for the goal of video processing; by performing transmission preprocessing on the original video, the applicability of the method is improved; by dividing the image blocks, the display position of the original video on the spliced ​​large screen is clarified, providing a reference for optimizing the display effect; by detecting the color state of the image block, a reference is provided for super-resolution super-frame reconstruction, reducing the amount of calculation for super-resolution super-frame reconstruction; by determining the super-resolution super-frame target, a reference is provided for super-resolution super-frame reconstruction, reducing the amount of calculation for super-resolution super-frame reconstruction; through the super-resolution super-frame algorithm, super-resolution super-frame reconstruction of the image block to be displayed is realized, thereby optimizing the display effect.

[0020] Preferably, the resolution-reduced frame rate processing and the inverse processing of the resolution-reduced frame rate processing specifically include:

[0021] The resolution reduction frame rate formula is used to process the original video that does not meet the low-bandwidth network transmission requirements. The resolution reduction frame rate formula is: Among them, LV is a low-resolution and low-frame-rate video, V is the original video, The function is a degradation mapping function of the original video degenerated into a video with low resolution and low frame rate, α is the first factor in the resolution degradation process, and β is the second factor in the frame rate degradation process;

[0022] Defining the low-resolution and low-frame-rate video as a first video;

[0023] Packing all frames generated by the original video during the frame rate degradation process and position data corresponding to all frames to obtain a frame rate degradation data packet, and transmitting the frame rate degradation data packet;

[0024] Receive the first video and the frame rate degradation data packet, parse the frame rate degradation data packet, obtain all frames generated in the frame rate degradation process of the original video and the position data corresponding to all frames, and degrade the original video into a video with low resolution and low frame rate based on the degradation mapping function Restore the first video to the original video.

[0025] Through the above, the low-bandwidth network transmission requirements are met by reducing the resolution and frame rate of the original video; by retaining the frame rate degraded data packets, data guarantee is provided for the accurate restoration of the original video; through the above transmission preprocessing of the original video, the applicability of the method is improved.

[0026] Preferably, step S5 specifically includes:

[0027] Using the number M of sub-screens in the horizontal direction of the spliced ​​large screen and the number N of sub-screens in the vertical direction of the spliced ​​large screen in the screen information, each frame in the original video is divided into M*N image blocks;

[0028] Using the sub-screen screen ratio in the screen information, setting the image block aspect ratio to be equal to the sub-screen screen ratio;

[0029] Each image block in each frame of the original video is mapped to the position of the corresponding sub-screen in the spliced ​​large screen according to its position in the frame.

[0030] Through the above, by dividing the image blocks and limiting the image blocks through screen information, and establishing the corresponding relationship between the image blocks and the sub-screens, accurate positioning is achieved when splicing large-screen displays, providing a reference for obtaining the video to be displayed; by limiting the aspect ratio of the image blocks, the foundation is laid for optimizing the image block processing.

[0031] Preferably, for any of the image blocks, detecting the color state of the image block specifically includes:

[0032] The hue type, the hue brightness corresponding to each color and the hue purity corresponding to each color in the image block are defined as a color state, and the color state is divided into a high-order color state and a low-order color state;

[0033] The color state of each image block is calculated using a color state formula, wherein the color state formula is ColorState=δ(type)+ε(brightness)+φ(purity), wherein δ() represents a judgment function of hue type, ε() represents a judgment function of hue brightness, and φ() represents a judgment formula of hue purity;

[0034] Preset hue type threshold type0 and hue type difference threshold γ;

[0035] The hue type difference is calculated using the hue type judgment function, where the hue type judgment function is:

[0036] Preset hue-brightness threshold value bright0 and hue-brightness total difference threshold value η;

[0037] The total hue-brightness difference is calculated using the hue-brightness judgment function. The hue type judgment function is: in, It represents the sum of the hue-lightness differences of each hue;

[0038] Preset the hue purity threshold value purity0 and the hue purity total difference threshold value κ;

[0039] The total hue purity difference is calculated using the hue purity judgment function, where the hue purity judgment function is: in, It represents the sum of the hue purity differences of each hue;

[0040] When the color state Color State of the image block is ≥ 2, the color state of the image block is defined as a high-order color state; otherwise, the color state of the image block is defined as a low-order color state.

[0041] Based on the above, by defining and calculating the color state, the classification of image blocks is achieved, which provides a basis for optimizing the image block processing.

[0042] Preferably, step S7 specifically includes:

[0043] Determine whether the resolution of the original video is greater than or equal to the sub-screen resolution, if yes, no super-resolution reconstruction is required, otherwise, determine the sub-screen resolution as the super-resolution target;

[0044] Determine whether the frame rate of the original video is greater than or equal to the sub-screen frame rate, if yes, no super-frame reconstruction is required, otherwise, determine the sub-screen frame rate as the super-frame target;

[0045] A super-resolution super-frame target is determined using the super-resolution target and the super-frame target.

[0046] By determining the super-resolution super-frame target, data support is provided for super-resolution super-weight reconstruction, thereby reducing the amount of computation for super-resolution super-frame reconstruction and achieving an optimized super-resolution low super-resolution super-frame reconstruction process.

[0047] Preferably, step S8 specifically includes:

[0048] A1: using the super-resolution super-frame target to determine whether the original video needs to be reconstructed by super-resolution or super-frame, if yes, execute step A2; otherwise, execute step A4;

[0049] A2: Determine whether the color state of the image block in the original video is a high-order color state, if yes, execute step A3, otherwise execute step A4;

[0050] A3: determining the image block as the image block to be reconstructed;

[0051] A4: There is no need to reconstruct the image block.

[0052] Based on the above, the image blocks that need to be reconstructed by super-resolution super-frame are identified based on the color state and the super-resolution super-frame target, and the image blocks that do not need to be reconstructed by super-resolution super-frame are screened out, thereby reducing the computational complexity of the super-resolution super-frame process.

[0053] Preferably, step S9 specifically includes:

[0054] B1: Determine whether the image block to be reconstructed needs to be super-reconstructed, if yes, execute step B2, otherwise execute step B3;

[0055] B2: Perform super-resolution reconstruction on the image block to be reconstructed based on a preset super-resolution algorithm to obtain a super-resolution image block, and determine whether the super-resolution image block needs to be super-frame reconstructed. If yes, execute step B4; otherwise, execute step B5

[0056] B3: Perform super-frame reconstruction between the image blocks to be reconstructed based on a preset super-frame algorithm to obtain super-frame image blocks, and execute step B5;

[0057] B4: performing super-frame reconstruction between the super-resolved image blocks based on a preset super-frame algorithm to obtain super-resolved super-frame image blocks, and executing step B5;

[0058] B5: define any one of the super-resolved image block, the super-frame image block or the super-resolved super-frame image block as an image block to be displayed.

[0059] Through the above, by accurately classifying the reconstructed image blocks, accurate processing of the reconstructed image blocks is achieved, reducing the amount of computation in the super-resolution super-frame reconstruction process; through super-resolution super-frame processing of the reconstructed image blocks, optimization of the display effect on the spliced ​​large screen is achieved.

[0060] Preferably, obtaining the video to be displayed by using the image block to be displayed specifically includes:

[0061] Based on the established correspondence between the image block and the sub-screen, the image block to be displayed is mapped to the position of the corresponding sub-screen in the spliced ​​large screen to obtain each frame to be displayed;

[0062] The video to be displayed is obtained by using each frame to be displayed.

[0063] Through the above, through the correspondence between the image blocks and the sub-screens, accurate reconstruction of the video to be displayed is achieved, and the display effect on the spliced ​​large screen is optimized.

[0064] In a second aspect, the present application discloses a video transmission and display device based on a super-resolution super-frame algorithm, which is applicable to a video transmission and display method based on a super-resolution super-frame algorithm as described in any one of the above, and the device comprises:

[0065] A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video transmission and display method based on the super-resolution superframe algorithm as described in any one of claims 1-8.

[0066] In a third aspect, the present application discloses a video transmission and display system based on a super-resolution super-frame algorithm, which is applicable to any one of the above-mentioned video transmission and display methods based on a super-resolution super-frame algorithm and the above-mentioned video transmission and display device based on a super-resolution super-frame algorithm, and the system comprises:

[0067] A large spliced ​​screen composed of M*N identical sub-screens, the large spliced ​​screen is used to display videos processed based on a super-resolution super-frame algorithm;

[0068] An original video information acquisition and judgment module, wherein the original video information includes the original video, the resolution of the original video and the frame rate of the original video; the original video information acquisition and judgment module is configured to acquire the original video information and judge whether the original video meets the low-bandwidth network transmission requirement, and if so, transmit the original video to the original video preprocessing module, otherwise, transmit the original video to the resolution reduction and frame reduction module;

[0069] The resolution reduction and frame reduction and restoration module is configured to: receive the original video that does not meet the low-bandwidth network transmission requirements, reduce the resolution and frame rate of the original video to obtain a first video; and transmit the first video to the original video preprocessing module;

[0070] An original video preprocessing module, wherein the original video preprocessing module is configured to: receive any one of the original video or the first video, perform inverse processing of reducing resolution and frame rate on the first video to restore the original video; obtain the screen information, divide each frame in the original video into multiple image blocks based on the screen information, and establish a corresponding relationship between the image blocks and the sub-screen;

[0071] An image block processing module, wherein the image block processing module is configured to detect a color state of any image block; the color state is used to determine whether the image block needs to be reconstructed;

[0072] A super-resolution super-frame processing module, wherein the super-resolution super-frame processing module is configured to: determine a super-resolution super-frame target based on the screen information; determine each image block to be reconstructed using the color state and the super-resolution super-frame target; and perform super-resolution super-frame reconstruction on each image block to be reconstructed based on a preset super-resolution super-frame algorithm to obtain an image block to be displayed;

[0073] The module for acquiring video to be displayed is configured to obtain the video to be displayed by using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

[0074] Based on the above, the technical effect of the video transmission and display method based on the super-resolution super-frame algorithm is achieved through the video transmission and display system based on the super-resolution super-frame algorithm.

[0075] Beneficial effects: The video transmission and display method, device and system based on the super-resolution superframe algorithm of the present application provide data support for the goal of video processing by obtaining the screen information of the spliced ​​large screen; improve the applicability of the method by performing transmission preprocessing on the original video; clarify the display position of the original video on the spliced ​​large screen by dividing the image blocks, and provide a reference for optimizing the display effect; provide a reference for super-resolution superframe reconstruction by detecting the color state of the image block, and reduce the amount of calculation of super-resolution superframe reconstruction; provide a reference for super-resolution superframe reconstruction by determining the super-resolution superframe target, and reduce the amount of calculation of super-resolution superframe reconstruction; realize super-resolution superframe reconstruction of the image block to be displayed through the super-resolution superframe algorithm, and optimize the display effect. Finally, the optimization of the video transmission and display process is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 A flowchart of a video transmission and display method based on a super-resolution super-frame algorithm provided in an embodiment of the present application;

[0078] Figure 2 A schematic diagram of dividing an image block and establishing a corresponding relationship between the image block and the sub-screen provided in an embodiment of the present application;

[0079] Figure 3 A schematic block diagram of a video transmission and display system based on a super-resolution super-frame algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0081] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0082] The first aspect of this embodiment discloses Figure 1 A video transmission and display method based on a super-resolution super-frame algorithm is shown, the method comprising:

[0083] S1: Build a spliced ​​large screen composed of M*N identical sub-screens, and obtain the sub-screen resolution, sub-screen frame rate and sub-screen screen ratio; wherein M is the number of sub-screens in the horizontal direction of the spliced ​​large screen, and N is the number of sub-screens in the vertical direction of the spliced ​​large screen; record screen information, which includes the number of sub-screens in the horizontal direction of the spliced ​​large screen, the number of sub-screens in the vertical direction of the spliced ​​large screen, the sub-screen resolution, the sub-screen frame rate and the sub-screen screen ratio.

[0084] S2: Obtain the original video and the corresponding resolution and frame rate, and determine whether the original video meets the low-bandwidth network transmission requirements. If so, transmit the original video and execute step S4; otherwise, reduce the resolution and frame rate of the original video to obtain the first video and execute step S3.

[0085] S3: Receive the first video, perform inverse processing of reducing the resolution and frame rate of the first video, restore the first video to the original video, transmit the original video and execute step S4.

[0086] S4: Receive the original video and obtain screen information.

[0087] S5: Divide each frame in the original video into multiple image blocks based on the screen information, and establish a corresponding relationship between the image blocks and the sub-screens.

[0088] S6: For any image block, detect the color state of the image block, and the color state is used to determine whether the image block needs to be reconstructed.

[0089] S7: Determine a super-resolution super-frame target based on the screen information.

[0090] S8: Determine each image block to be reconstructed using the color state and the super-resolution super-frame target.

[0091] S9: Based on a preset super-resolution super-frame algorithm, super-resolution super-frame reconstruction is performed on each image block to be reconstructed to obtain an image block to be displayed.

[0092] S10: Obtain the video to be displayed using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

[0093] By acquiring the screen information of the spliced ​​large screen, data support is provided for the goal of video processing; by performing transmission preprocessing on the original video, the applicability of the method is improved; by dividing the image blocks, the display position of the original video on the spliced ​​large screen is clarified, providing a reference for optimizing the display effect; by detecting the color state of the image block, a reference is provided for super-resolution super-frame reconstruction, reducing the amount of calculation for super-resolution super-frame reconstruction; by determining the super-resolution super-frame target, a reference is provided for super-resolution super-frame reconstruction, reducing the amount of calculation for super-resolution super-frame reconstruction; and by using the super-resolution super-frame algorithm, super-resolution super-frame reconstruction of the image block to be displayed is realized, thereby optimizing the display effect.

[0094] Specifically, the resolution frame rate reduction processing and the inverse processing of the resolution frame rate reduction processing specifically include:

[0095] The resolution reduction frame rate formula is used to process the original video that does not meet the low-bandwidth network transmission requirements. The resolution reduction frame rate formula is: Among them, LV is a low-resolution and low-frame-rate video, V is the original video, The function is a degradation mapping function of the original video degenerated into a video with low resolution and low frame rate, α is the first factor in the resolution degradation process, and β is the second factor in the frame rate degradation process.

[0096] It should be noted that the first factor and the second factor may be rules set artificially in the resolution degradation process and the frame rate degradation process, such as sampling frequency.

[0097] A video with low resolution and low frame rate is defined as the first video.

[0098] All frames generated during the frame rate degradation process of the original video and position data corresponding to all frames are packaged to obtain a frame rate degradation data packet, and the frame rate degradation data packet is transmitted.

[0099] Receive the first video and the frame rate degradation data packet, parse the frame rate degradation data packet, obtain all frames generated in the frame rate degradation process of the original video and the position data corresponding to all frames, and calculate the degradation mapping function based on the degradation of the original video into a video with low resolution and low frame rate Restore the first video to the original video.

[0100] By reducing the resolution and frame rate of the original video, the low-bandwidth network transmission requirements are met; by retaining the frame rate degraded data packets, data guarantee is provided for the accurate restoration of the original video; through the above-mentioned transmission preprocessing of the original video, the applicability of the method is improved.

[0101] Specifically, step S5 specifically includes:

[0102] Using the number M of sub-screens in the horizontal direction of the spliced ​​large screen and the number N of sub-screens in the vertical direction of the spliced ​​large screen in the screen information, each frame in the original video is divided into M*N image blocks;

[0103] The image block aspect ratio is set to be equal to the sub-screen screen ratio by using the sub-screen screen ratio in the screen information.

[0104] Each image block in each frame of the original video is mapped to the position of the corresponding sub-screen in the spliced ​​large screen according to its position in the frame.

[0105] As a simple example, combining Figure 2 As shown, the right side is a spliced ​​large screen, in which each rectangular box is a sub-screen, and there are 2*2 sub-screens in total. The left side is a frame in the original video; according to the sub-screen on the right, the frame on the left is divided into four image blocks of equal proportion to the sub-screen, and each color represents an image block; the arrow indicates the mapping method, and the part of the sub-screen with the same color as the image block indicates the display position of the image block in the spliced ​​large screen.

[0106] By dividing image blocks and limiting image blocks through screen information and establishing a corresponding relationship between image blocks and sub-screens, accurate positioning is achieved when splicing large-screen displays, providing a reference for obtaining the video to be displayed; by limiting the aspect ratio of image blocks, the foundation is laid for optimizing image block processing.

[0107] Specifically, for any image block, detecting the color state of the image block specifically includes:

[0108] The hue type in the image block, the hue brightness corresponding to each color, and the hue purity corresponding to each color are defined as a color state, and the color state is divided into two states: a high-order color state and a low-order color state.

[0109] The color state of each image block is calculated using the color state formula, which is Color State = δ(type) + ε(brightness) + φ(purity), where δ() represents the judgment function of the hue type, ε() represents the judgment function of the hue brightness, and φ() represents the judgment formula of the hue purity.

[0110] A hue type threshold type0 and a hue type difference threshold γ are preset.

[0111] The hue type difference is calculated using the hue type judgment function. The hue type judgment function is:

[0112]

[0113] It should be noted that the hue type difference evaluates the richness of the hue in the image block, and its type is proportional to the richness.

[0114] In a simple example, an image block has only one hue, such as a frame with a title written on a white background in a video. In this case, there is no need to perform super-resolution super-frame reconstruction on the part with the white background.

[0115] The hue-brightness threshold value bright0 and the hue-brightness total difference threshold value η are preset.

[0116] The total hue-brightness difference is calculated using the hue-brightness judgment function. The hue type judgment function is: in, Represents the sum of the hue-lightness differences of each hue.

[0117] It should be noted that the total hue-brightness difference evaluates the sum of the brightness differences of each hue of the image block, which evaluates the brightness of the image block as a whole; the hue-brightness threshold is a restriction on the brightness of each hue, which can be more than one based on the screen information obtained from the sub-screen; each hue brightness may have a negative value, indicating that the brightness of the hue in the image block is less than the hue-brightness threshold of the hue, and the summation calculation is performed directly without taking the absolute value.

[0118] The hue purity threshold purity0 and the hue purity total difference threshold κ are preset.

[0119] The total hue purity difference is calculated using the hue purity judgment function. The hue purity judgment function is: in, Represents the sum of the hue purity differences of each hue.

[0120] It should be noted that the total hue purity difference evaluates the sum of the purity differences of each hue of the image block, which evaluates the purity of the image block as a whole; the hue purity threshold is a restriction on the purity of each hue, which can be more than one based on the screen information obtained from the sub-screen; each hue purity may have a negative value, indicating that the purity of the hue in the image block is less than the hue purity threshold of the hue, and the summation calculation is performed directly without taking the absolute value.

[0121] When the color state Color State of the image block is ≥ 2, the color state of the image block is defined as a high-order color state, otherwise the color state of the image block is defined as a low-order color state.

[0122] It should be noted that the result calculated based on the color state formula is any one of 0, 1, 2 and 3. When the color state of the image block is Color State≥2, the color state of the image block can be accurately divided.

[0123] By defining and calculating the color states, the classification of image blocks is achieved, providing a basis for optimizing image block processing.

[0124] Specifically, step S7 specifically includes:

[0125] Determine whether the resolution of the original video is greater than or equal to the sub-screen resolution. If so, no super-resolution reconstruction is required. Otherwise, the sub-screen resolution is determined as the super-resolution target.

[0126] Determine whether the frame rate of the original video is greater than or equal to the sub-screen frame rate. If so, no super-frame reconstruction is required. Otherwise, the sub-screen frame rate is determined as the super-frame target.

[0127] The super-resolution target and the super-frame target are used to determine the super-resolution super-frame target.

[0128] It should be noted that step S7 divides the original video into three types: one that only needs super-resolution reconstruction, one that only needs super-frame reconstruction, and one that needs both super-resolution reconstruction and super-frame reconstruction. This is the basis for reducing the amount of calculation.

[0129] By determining the super-resolution super-frame target, data support is provided for super-resolution and super-weight reconstruction, thereby reducing the amount of calculation for super-resolution super-frame reconstruction and achieving the optimization of super-resolution and low-resolution super-frame reconstruction process.

[0130] Specifically, step S8 specifically includes:

[0131] A1: using the super-resolution super-frame target to determine whether the original video needs at least one of super-resolution reconstruction or super-frame reconstruction, if yes, execute step A2, otherwise execute step A4;

[0132] A2: Determine whether the color state of the image block in the original video is a high-order color state, if yes, execute step A3, otherwise execute step A4;

[0133] A3: determining the image block as the image block to be reconstructed;

[0134] A4: No need to reconstruct the image blocks.

[0135] It should be noted that, based on step S7, step S8 further screens the image blocks of each frame of the original video based on the color status of the image blocks, thereby further providing a basis for reducing the amount of calculation.

[0136] Based on the color state and super-resolution superframe target, the image blocks that need to be reconstructed by super-resolution superframe are identified, and the image blocks that do not need to be reconstructed by super-resolution superframe are screened out, thereby reducing the computational complexity of the super-resolution superframe process.

[0137] Specifically, step S9 specifically includes:

[0138] B1: Determine whether the image block to be reconstructed needs to be super-reconstructed, if yes, execute step B2, otherwise execute step B3;

[0139] B2: Perform super-resolution reconstruction on the image block to be reconstructed based on the preset super-resolution algorithm to obtain a super-resolution image block, and determine whether the super-resolution image block needs to be super-frame reconstructed. If yes, execute step B4; otherwise, execute step B5

[0140] B3: Perform super-frame reconstruction on the image blocks to be reconstructed based on a preset super-frame algorithm to obtain super-frame image blocks, and execute step B5;

[0141] B4: Perform super-frame reconstruction between super-resolved image blocks based on a preset super-frame algorithm to obtain super-resolved super-frame image blocks, and execute step B5;

[0142] B5: Define any one of the super-resolved image block, the super-frame image block or the super-resolved super-frame image block as the image block to be displayed.

[0143] It should be noted that the super-resolution algorithm and super-frame algorithm in this embodiment can be any one of the existing technologies; step S9, based on steps S7 and S8, further divides the image blocks to be reconstructed into three types: those that only require super-resolution reconstruction, those that only require super-frame reconstruction, and those that require both super-resolution reconstruction and super-frame reconstruction. This is the basis for reducing the amount of computation.

[0144] By accurately classifying the image blocks to be reconstructed, precise processing of the image blocks to be reconstructed is achieved, reducing the amount of computation in the super-resolution and super-frame reconstruction process; by super-resolution and super-frame processing of the reconstructed image blocks, optimization of the display effect on the spliced ​​large screen is achieved.

[0145] Specifically, obtaining the video to be displayed by using the image block to be displayed specifically includes:

[0146] Based on the established correspondence between the image blocks and the sub-screens, the image blocks to be displayed are mapped to the positions of the corresponding sub-screens in the spliced ​​large screen to obtain each frame to be displayed.

[0147] The video to be displayed is obtained by using each frame to be displayed.

[0148] It should be noted that the spliced ​​large screen is different from the traditional large screen. It is composed of multiple sub-screens. Due to the asynchrony of the image block processing process and the mapping errors that may occur during the display process, the sub-screens may be out of sync during the display process. Establishing the correspondence between the image blocks and the sub-screens in advance and reducing the amount of calculation can effectively avoid the occurrence of this phenomenon.

[0149] Through the correspondence between the image blocks and the sub-screens, accurate reconstruction of the video to be displayed is achieved, and the display effect on the spliced ​​large screen is optimized.

[0150] A second aspect of the present embodiment discloses a video transmission and display device based on a super-resolution super-frame algorithm, which is applicable to any of the above-mentioned video transmission and display methods based on a super-resolution super-frame algorithm, and the device includes:

[0151] A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the video transmission and display method based on the super-resolution super-frame algorithm as claimed in any one of claims 1 to 8.

[0152] The third aspect of this embodiment discloses Figure 3 A video transmission and display system based on a super-resolution super-frame algorithm is shown, which is applicable to any of the above-mentioned video transmission and display methods based on a super-resolution super-frame algorithm and the above-mentioned video transmission and display device based on a super-resolution super-frame algorithm, and the system comprises:

[0153] A large splicing screen composed of M*N identical sub-screens is used to display videos processed based on super-resolution and super-frame algorithms;

[0154] The original video information acquisition and judgment module includes the original video, the resolution of the original video and the frame rate of the original video; the original video information acquisition and judgment module is configured to acquire the original video information and judge whether the original video meets the low-bandwidth network transmission requirements, if yes, the original video is transmitted to the original video preprocessing module, otherwise, the original video is transmitted to the resolution reduction and frame reduction module;

[0155] The resolution reduction and frame reduction and restoration module is configured to: receive the original video that does not meet the low-bandwidth network transmission requirements, reduce the resolution and frame rate of the original video to obtain a first video; and transmit the first video to the original video preprocessing module;

[0156] The original video preprocessing module is configured to: receive any one of the original video or the first video, perform inverse processing of reducing the resolution and frame rate of the first video to restore the original video; obtain screen information, divide each frame in the original video into multiple image blocks based on the screen information, and establish a corresponding relationship between the image blocks and the sub-screens;

[0157] An image block processing module, the image block processing module is configured to detect the color state of any image block; the color state is used to determine whether the image block needs to be reconstructed;

[0158] The super-resolution super-frame processing module is configured to: determine the super-resolution super-frame target based on the screen information; determine each image block to be reconstructed by using the color state and the super-resolution super-frame target; perform super-resolution super-frame reconstruction on each image block to be reconstructed based on a preset super-resolution super-frame algorithm to obtain an image block to be displayed;

[0159] The module for acquiring the video to be displayed is configured to obtain the video to be displayed by using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

[0160] The technical effect of the video transmission and display method based on the super-resolution super-frame algorithm is achieved through the video transmission and display system based on the super-resolution super-frame algorithm.

[0161] In summary, the video transmission and display method, device and system based on the super-resolution superframe algorithm of this embodiment provides data support for the goal of video processing by obtaining the screen information of the spliced ​​large screen; improves the applicability of the method by performing transmission preprocessing on the original video; clarifies the display position of the original video on the spliced ​​large screen by dividing the image blocks, providing a reference for optimizing the display effect; provides a reference for super-resolution superframe reconstruction by detecting the color state of the image block, reducing the amount of calculation for super-resolution superframe reconstruction; provides a reference for super-resolution superframe reconstruction by determining the super-resolution superframe target, reducing the amount of calculation for super-resolution superframe reconstruction; realizes super-resolution superframe reconstruction of the image block to be displayed through the super-resolution superframe algorithm, optimizing the display effect. Finally, the optimization of the video transmission and display process is achieved.

[0162] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.

[0163] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video transmission and display method based on super-resolution super-frame algorithm, characterized in that: The method includes: S1: Build a spliced ​​large screen consisting of M*N identical sub-screens, and obtain the sub-screen resolution, sub-screen frame rate, and sub-screen screen ratio; wherein M is the number of sub-screens in the horizontal direction of the spliced ​​large screen, and N is the number of sub-screens in the vertical direction of the spliced ​​large screen; record screen information, which includes the number of sub-screens in the horizontal direction of the spliced ​​large screen, the number of sub-screens in the vertical direction of the spliced ​​large screen, the sub-screen resolution, the sub-screen frame rate, and the sub-screen screen ratio; S2: Obtain the original video and the corresponding resolution and frame rate, and determine whether the original video meets the low-bandwidth network transmission requirement. If yes, transmit the original video and execute step S4; otherwise, reduce the resolution and frame rate of the original video to obtain the first video and execute step S3; S3: receiving the first video, performing inverse processing of reducing the resolution and frame rate of the first video, restoring the first video to the original video, transmitting the original video and executing step S4; S4: receiving the original video and acquiring the screen information; S5: dividing each frame in the original video into a plurality of image blocks based on the screen information, and establishing a corresponding relationship between the image blocks and the sub-screens; S6: for any of the image blocks, detecting a color state of the image block, wherein the color state is used to determine whether the image block needs to be reconstructed; S7: Determine a super-resolution super-frame target based on the screen information; S8: Determine each image block to be reconstructed by using the color state and the super-resolution super-frame target; S9: Based on a preset super-resolution super-frame algorithm, super-resolution super-frame reconstruction is performed on each of the image blocks to be reconstructed to obtain an image block to be displayed; S10: Obtain a video to be displayed using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

2. The video transmission and display method based on super-resolution super-frame algorithm according to claim 1 is characterized in that: The resolution-reduced frame rate processing and the inverse processing of the resolution-reduced frame rate processing specifically include: The resolution reduction frame rate formula is used to process the original video that does not meet the low-bandwidth network transmission requirements. The resolution reduction frame rate formula is: Among them, LV is a low-resolution and low-frame-rate video, V is the original video, The function is a degradation mapping function of the original video degenerated into a video with low resolution and low frame rate, α is the first factor in the resolution degradation process, and β is the second factor in the frame rate degradation process; Defining the low-resolution and low-frame-rate video as a first video; Packing all frames generated by the original video during the frame rate degradation process and position data corresponding to all frames to obtain a frame rate degradation data packet, and transmitting the frame rate degradation data packet; Receive the first video and the frame rate degradation data packet, parse the frame rate degradation data packet, obtain all frames generated in the frame rate degradation process of the original video and the position data corresponding to all frames, and degrade the original video into a video with low resolution and low frame rate based on the degradation mapping function Restore the first video to the original video.

3. The video transmission and display method based on super-resolution super-frame algorithm according to claim 1 is characterized in that: Step S5 specifically includes: Using the number M of sub-screens in the horizontal direction of the spliced ​​large screen and the number N of sub-screens in the vertical direction of the spliced ​​large screen in the screen information, each frame in the original video is divided into M*N image blocks; Using the sub-screen screen ratio in the screen information, setting the image block aspect ratio to be equal to the sub-screen screen ratio; Each image block in each frame of the original video is mapped to the position of the corresponding sub-screen in the spliced ​​large screen according to its position in the frame.

4. The video transmission and display method based on super-resolution super-frame algorithm according to claim 3 is characterized in that: The detecting the color state of any of the image blocks specifically includes: The hue type, the hue brightness corresponding to each color and the hue purity corresponding to each color in the image block are defined as a color state, and the color state is divided into a high-order color state and a low-order color state; The color state of each image block is calculated using a color state formula, wherein the color state formula is Color State = δ(type) + ε(brightness) + φ(purity), wherein δ() represents a judgment function of hue type, ε() represents a judgment function of hue brightness, and φ() represents a judgment formula of hue purity; Preset hue type threshold type0 and hue type difference threshold γ; The hue type difference is calculated using the hue type judgment function, where the hue type judgment function is: Preset hue-brightness threshold value bright0 and hue-brightness total difference threshold value η; The total hue-brightness difference is calculated using the hue-brightness judgment function. The hue type judgment function is: in, It represents the sum of the hue-lightness differences of each hue; Preset the hue purity threshold value purity0 and the hue purity total difference threshold value κ; The total hue purity difference is calculated using the hue purity judgment function, where the hue purity judgment function is: in, It represents the sum of the hue purity differences of each hue; When the color state Color State of the image block is ≥ 2, the color state of the image block is defined as a high-order color state; otherwise, the color state of the image block is defined as a low-order color state.

5. The video transmission and display method based on super-resolution super-frame algorithm according to claim 4 is characterized in that: Step S7 specifically includes: Determine whether the resolution of the original video is greater than or equal to the sub-screen resolution, if yes, no super-resolution reconstruction is required, otherwise, determine the sub-screen resolution as the super-resolution target; Determine whether the frame rate of the original video is greater than or equal to the sub-screen frame rate, if yes, no super-frame reconstruction is required, otherwise, determine the sub-screen frame rate as the super-frame target; A super-resolution super-frame target is determined using the super-resolution target and the super-frame target.

6. The video transmission and display method based on super-resolution super-frame algorithm according to claim 5 is characterized in that: Step S8 specifically includes: A1: using the super-resolution super-frame target to determine whether the original video needs to be reconstructed by super-resolution or super-frame, if yes, execute step A2; otherwise, execute step A4; A2: Determine whether the color state of the image block in the original video is a high-order color state, if yes, execute step A3, otherwise execute step A4; A3: determining the image block as the image block to be reconstructed; A4: There is no need to reconstruct the image block.

7. The video transmission and display method based on super-resolution super-frame algorithm according to claim 6 is characterized in that: Step S9 specifically includes: B1: Determine whether the image block to be reconstructed needs to be super-reconstructed, if yes, execute step B2, otherwise execute step B3; B2: Perform super-resolution reconstruction on the image block to be reconstructed based on a preset super-resolution algorithm to obtain a super-resolution image block, and determine whether the super-resolution image block needs to be super-frame reconstructed. If yes, execute step B4; otherwise, execute step B5 B3: Perform super-frame reconstruction between the image blocks to be reconstructed based on a preset super-frame algorithm to obtain super-frame image blocks, and execute step B5; B4: performing super-frame reconstruction between the super-resolved image blocks based on a preset super-frame algorithm to obtain super-resolved super-frame image blocks, and executing step B5; B5: define any one of the super-resolved image block, the super-frame image block or the super-resolved super-frame image block as an image block to be displayed.

8. The video transmission and display method based on super-resolution super-frame algorithm according to claim 7 is characterized in that: The step of obtaining the video to be displayed by using the image block to be displayed specifically includes: Based on the established correspondence between the image block and the sub-screen, the image block to be displayed is mapped to the position of the corresponding sub-screen in the spliced ​​large screen to obtain each frame to be displayed; The video to be displayed is obtained by using each frame to be displayed.

9. A video transmission and display device based on super-resolution super-frame algorithm, applicable to the video transmission and display method based on super-resolution super-frame algorithm as claimed in any one of claims 1 to 8, characterized in that: The device includes: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video transmission and display method based on the super-resolution superframe algorithm as described in any one of claims 1-8.

10. A video transmission and display system based on super-resolution super-frame algorithm, applicable to the video transmission and display method based on super-resolution super-frame algorithm according to any one of claims 1 to 8 and the video transmission and display device based on super-resolution super-frame algorithm according to claim 9, characterized in that: The system includes: A large spliced ​​screen composed of M*N identical sub-screens, the large spliced ​​screen is used to display videos processed based on a super-resolution super-frame algorithm; An original video information acquisition and judgment module, wherein the original video information includes the original video, the resolution of the original video and the frame rate of the original video; the original video information acquisition and judgment module is configured to acquire the original video information and judge whether the original video meets the low-bandwidth network transmission requirement, and if so, transmit the original video to the original video preprocessing module, otherwise, transmit the original video to the resolution reduction and frame reduction module; The resolution reduction and frame reduction and restoration module is configured to: receive the original video that does not meet the low-bandwidth network transmission requirements, reduce the resolution and frame rate of the original video to obtain a first video; and transmit the first video to the original video preprocessing module; An original video preprocessing module, wherein the original video preprocessing module is configured to: receive any one of the original video or the first video, perform inverse processing of reducing resolution and frame rate on the first video to restore the original video; obtain the screen information, divide each frame in the original video into multiple image blocks based on the screen information, and establish a corresponding relationship between the image blocks and the sub-screen; An image block processing module, wherein the image block processing module is configured to detect a color state of any image block; the color state is used to determine whether the image block needs to be reconstructed; A super-resolution super-frame processing module, wherein the super-resolution super-frame processing module is configured to: determine a super-resolution super-frame target based on the screen information; determine each image block to be reconstructed using the color state and the super-resolution super-frame target; and perform super-resolution super-frame reconstruction on each image block to be reconstructed based on a preset super-resolution super-frame algorithm to obtain an image block to be displayed; The module for acquiring video to be displayed is configured to obtain the video to be displayed by using the image blocks to be displayed, and display the video to be displayed on the spliced ​​large screen.

Images